mirrors/ladybird
1
0
Fork 0
mirror of https://github.com/LadybirdBrowser/ladybird.git synced 2025-01-24 10:12:25 -05:00
Code Issues Projects Releases Packages Wiki Activity
ladybird/Meta/CMake/unicode_data.cmake

198 lines
8.8 KiB
CMake
Raw Normal View History

Meta: Move remove_unicode_data_if_version_changed to utils.cmake This function will be used by the time zone database parser. Move it to the common utilities file, and rename it remove_path_if_version_changed to be more generic.
2021-12-22 16:12:59 -05:00
include(${CMAKE_CURRENT_LIST_DIR}/utils.cmake)
LibUnicode: Upgrade to Unicode version 14.0.0
2021-09-15 09:35:18 -04:00
set(UCD_VERSION 14.0.0)
LibUnicode: Upgrade to CLDR version 40.0.0 Release notes: https://github.com/unicode-org/cldr-json/releases/tag/40.0.0
2021-11-09 10:50:37 -05:00
set(CLDR_VERSION 40.0.0)
LibUnicode: Extract UCD and CLDR versions to a CMake variable This also surrounds expansion of affected URL and path variables with quotes.
2021-09-15 08:46:27 -04:00
LibUnicode: Use consistent variable naming in unicode_data.cmake I kept mixing these up in my head.
2021-09-15 13:27:17 -04:00
set(UCD_PATH "${CMAKE_BINARY_DIR}/UCD" CACHE PATH "Download location for UCD files")
set(CLDR_PATH "${CMAKE_BINARY_DIR}/CLDR" CACHE PATH "Download location for CLDR files")
LibUnicode: Introduce a Unicode library for interacting with UCD files The Unicode standard publishes the Unicode Character Database (UCD) with information about every code point, such as each code point's upper case mapping. LibUnicode exists to download and parse UCD files at build time and to provide accessors to that data. As a start, LibUnicode includes upper- and lower-case code point converters.
2021-07-25 15:10:51 -04:00
LibUnicode: Remove cached UCD and CLDR files when their version changes
2021-09-15 09:33:55 -04:00
set(UCD_VERSION_FILE "${UCD_PATH}/version.txt")
set(CLDR_VERSION_FILE "${CLDR_PATH}/version.txt")
LibUnicode: Extract UCD and CLDR versions to a CMake variable This also surrounds expansion of affected URL and path variables with quotes.
2021-09-15 08:46:27 -04:00
set(UNICODE_DATA_URL "https://www.unicode.org/Public/${UCD_VERSION}/ucd/UnicodeData.txt")
LibUnicode: Use consistent variable naming in unicode_data.cmake I kept mixing these up in my head.
2021-09-15 13:27:17 -04:00
set(UNICODE_DATA_PATH "${UCD_PATH}/UnicodeData.txt")
LibUnicode: Introduce a Unicode library for interacting with UCD files The Unicode standard publishes the Unicode Character Database (UCD) with information about every code point, such as each code point's upper case mapping. LibUnicode exists to download and parse UCD files at build time and to provide accessors to that data. As a start, LibUnicode includes upper- and lower-case code point converters.
2021-07-25 15:10:51 -04:00
LibUnicode: Extract UCD and CLDR versions to a CMake variable This also surrounds expansion of affected URL and path variables with quotes.
2021-09-15 08:46:27 -04:00
set(SPECIAL_CASING_URL "https://www.unicode.org/Public/${UCD_VERSION}/ucd/SpecialCasing.txt")
LibUnicode: Use consistent variable naming in unicode_data.cmake I kept mixing these up in my head.
2021-09-15 13:27:17 -04:00
set(SPECIAL_CASING_PATH "${UCD_PATH}/SpecialCasing.txt")
LibUnicode: Download and parse the special casing UCD file This adds a SpecialCasing structure to the generated UnicodeData.h/cpp files. This structure contains casing rules for code points which have non-1-to-1 upper-to-lower case code point mappings. Further, these rules may be limited to specific locales or other context.
2021-07-27 10:39:37 -04:00
LibUnicode: Extract UCD and CLDR versions to a CMake variable This also surrounds expansion of affected URL and path variables with quotes.
2021-09-15 08:46:27 -04:00
set(DERIVED_GENERAL_CATEGORY_URL "https://www.unicode.org/Public/${UCD_VERSION}/ucd/extracted/DerivedGeneralCategory.txt")
LibUnicode: Use consistent variable naming in unicode_data.cmake I kept mixing these up in my head.
2021-09-15 13:27:17 -04:00
set(DERIVED_GENERAL_CATEGORY_PATH "${UCD_PATH}/DerivedGeneralCategory.txt")
LibUnicode: Generate separate tables for General Category properties Previously, each code point's General Category was part of the generated UnicodeData structure. This ultimately presented two problems, one functional and one performance related: * Some General Categories are applied to unassigned code points, for example the Unassigned (Cn) category. Unassigned code points are strictly excluded from UnicodeData.txt, so by relying on that file, the generator is unable to handle these categories. * Lookups for General Categories are slower when searching through the large UnicodeData hash map. Even though lookups are O(1), the hash function turned out to be slower than binary searching through a category-specific table. So, now a table is generated for each General Category. When querying a code point for a category, a binary search is done on each code point range in that category's table to check if code point has that category. Further, General Categories are now parsed from the UCD file DerivedGeneralCategory.txt. This file is a normal "prop list" file and contains the categories for unassigned code points.
2021-08-10 07:17:24 -04:00
LibUnicode: Extract UCD and CLDR versions to a CMake variable This also surrounds expansion of affected URL and path variables with quotes.
2021-09-15 08:46:27 -04:00
set(PROP_LIST_URL "https://www.unicode.org/Public/${UCD_VERSION}/ucd/PropList.txt")
LibUnicode: Use consistent variable naming in unicode_data.cmake I kept mixing these up in my head.
2021-09-15 13:27:17 -04:00
set(PROP_LIST_PATH "${UCD_PATH}/PropList.txt")
LibUnicode: Download and parse the property list UCD file
2021-07-27 17:37:47 -04:00
LibUnicode: Extract UCD and CLDR versions to a CMake variable This also surrounds expansion of affected URL and path variables with quotes.
2021-09-15 08:46:27 -04:00
set(DERIVED_CORE_PROP_URL "https://www.unicode.org/Public/${UCD_VERSION}/ucd/DerivedCoreProperties.txt")
LibUnicode: Use consistent variable naming in unicode_data.cmake I kept mixing these up in my head.
2021-09-15 13:27:17 -04:00
set(DERIVED_CORE_PROP_PATH "${UCD_PATH}/DerivedCoreProperties.txt")
LibUnicode: Parse and utilize DerivedCoreProperties DerivedCoreProperties are pseudo-properties that are the union of other categories and properties. For example, the derived property Math is the union of the general category Sm and the property Other_Math. Parsing these is necessary for implementing Unicode property escapes. But it also has the added benefit that LibUnicode now does not need to derive some of these properties at runtime.
2021-07-28 18:39:41 -04:00
LibUnicode: Extract UCD and CLDR versions to a CMake variable This also surrounds expansion of affected URL and path variables with quotes.
2021-09-15 08:46:27 -04:00
set(DERIVED_BINARY_PROP_URL "https://www.unicode.org/Public/${UCD_VERSION}/ucd/extracted/DerivedBinaryProperties.txt")
LibUnicode: Use consistent variable naming in unicode_data.cmake I kept mixing these up in my head.
2021-09-15 13:27:17 -04:00
set(DERIVED_BINARY_PROP_PATH "${UCD_PATH}/DerivedBinaryProperties.txt")
LibUnicode: Parse UCD DerivedBinaryProperties.txt and generate property
2021-08-04 07:53:24 -04:00
LibUnicode: Extract UCD and CLDR versions to a CMake variable This also surrounds expansion of affected URL and path variables with quotes.
2021-09-15 08:46:27 -04:00
set(PROP_ALIAS_URL "https://www.unicode.org/Public/${UCD_VERSION}/ucd/PropertyAliases.txt")
LibUnicode: Use consistent variable naming in unicode_data.cmake I kept mixing these up in my head.
2021-09-15 13:27:17 -04:00
set(PROP_ALIAS_PATH "${UCD_PATH}/PropertyAliases.txt")
LibUnicode: Parse and generate PropertyAliases These are all used for Unicode property escapes.
2021-07-28 20:14:42 -04:00
LibUnicode: Extract UCD and CLDR versions to a CMake variable This also surrounds expansion of affected URL and path variables with quotes.
2021-09-15 08:46:27 -04:00
set(PROP_VALUE_ALIAS_URL "https://www.unicode.org/Public/${UCD_VERSION}/ucd/PropertyValueAliases.txt")
LibUnicode: Use consistent variable naming in unicode_data.cmake I kept mixing these up in my head.
2021-09-15 13:27:17 -04:00
set(PROP_VALUE_ALIAS_PATH "${UCD_PATH}/PropertyValueAliases.txt")
LibUnicode: Generate General Category unions and aliases This downloads the PropertyValueAliases.txt UCD file, which contains a set of General Category aliases. This changes the General Category enumeration to now be generated as a bitmask. This is to easily allow General Category unions. For example, the LC (Cased_Letter) category is the union of the Ll, Lu, and Lt categories.
2021-07-31 13:45:05 -04:00
Meta+LibUnicode: Provide code point names through library
2021-11-18 23:23:58 +01:00
set(NAME_ALIAS_URL "https://www.unicode.org/Public/${UCD_VERSION}/ucd/NameAliases.txt")
set(NAME_ALIAS_PATH "${UCD_PATH}/NameAliases.txt")
LibUnicode: Extract UCD and CLDR versions to a CMake variable This also surrounds expansion of affected URL and path variables with quotes.
2021-09-15 08:46:27 -04:00
set(SCRIPTS_URL "https://www.unicode.org/Public/${UCD_VERSION}/ucd/Scripts.txt")
LibUnicode: Use consistent variable naming in unicode_data.cmake I kept mixing these up in my head.
2021-09-15 13:27:17 -04:00
set(SCRIPTS_PATH "${UCD_PATH}/Scripts.txt")
LibUnicode: Parse UCD Scripts.txt and generate as a Unicode property There are a couple of minor nuances with parsing script values, compared to other properties. In Scripts.txt, the UCD file lists the full name of each script; other properties, like General Category, list the shorter name in their primary files. This means that the aliases listed in PropertyValueAliases.txt are reversed for script values.
2021-08-03 17:11:19 -04:00
LibUnicode: Extract UCD and CLDR versions to a CMake variable This also surrounds expansion of affected URL and path variables with quotes.
2021-09-15 08:46:27 -04:00
set(SCRIPT_EXTENSIONS_URL "https://www.unicode.org/Public/${UCD_VERSION}/ucd/ScriptExtensions.txt")
LibUnicode: Use consistent variable naming in unicode_data.cmake I kept mixing these up in my head.
2021-09-15 13:27:17 -04:00
set(SCRIPT_EXTENSIONS_PATH "${UCD_PATH}/ScriptExtensions.txt")
LibUnicode: Parse UCD ScriptExtensions.txt and generate property
2021-08-04 07:05:30 -04:00
LibUnicode: Extract UCD and CLDR versions to a CMake variable This also surrounds expansion of affected URL and path variables with quotes.
2021-09-15 08:46:27 -04:00
set(EMOJI_DATA_URL "https://www.unicode.org/Public/${UCD_VERSION}/ucd/emoji/emoji-data.txt")
LibUnicode: Use consistent variable naming in unicode_data.cmake I kept mixing these up in my head.
2021-09-15 13:27:17 -04:00
set(EMOJI_DATA_PATH "${UCD_PATH}/emoji-data.txt")
LibUnicode: Parse UCD emoji-data.txt and generate Unicode property
2021-08-04 07:46:36 -04:00
LibUnicode: Extract UCD and CLDR versions to a CMake variable This also surrounds expansion of affected URL and path variables with quotes.
2021-09-15 08:46:27 -04:00
set(NORM_PROPS_URL "https://www.unicode.org/Public/${UCD_VERSION}/ucd/DerivedNormalizationProps.txt")
LibUnicode: Use consistent variable naming in unicode_data.cmake I kept mixing these up in my head.
2021-09-15 13:27:17 -04:00
set(NORM_PROPS_PATH "${UCD_PATH}/DerivedNormalizationProps.txt")
LibUnicode: Download and parse DerivedNormalizationProps UCD file This file contains the last properties that LibUnicode is not parsing. Much of the data in this file is not currently used; that is left as a FIXME for when String.prototype.normalize is implemented. Until then, only the code point properties are utilized for regular expression pattern escapes.
2021-08-10 15:29:28 -04:00
LibUnicode: Extract UCD and CLDR versions to a CMake variable This also surrounds expansion of affected URL and path variables with quotes.
2021-09-15 08:46:27 -04:00
set(CLDR_ZIP_URL "https://github.com/unicode-org/cldr-json/releases/download/${CLDR_VERSION}/cldr-${CLDR_VERSION}-json-modern.zip")
LibUnicode: Use consistent variable naming in unicode_data.cmake I kept mixing these up in my head.
2021-09-15 13:27:17 -04:00
set(CLDR_ZIP_PATH "${CLDR_PATH}/cldr.zip")
LibUnicode: Download Unicode's CLDR database and generate locale data The Unicode standard publishes a database known as the Common Locale Data Repository (CLDR). This is a massive set of data from which anyone implementing Unicode's Technical Standard #35 may generate their implementation: https://www.unicode.org/reports/tr35/ This commit updates LibUnicode to download the compressed database and extract a small subset. That subset is used to generate a list of available locales and the territories (AKA regions) associated with each locale.
2021-08-24 19:50:18 -04:00
LibUnicode: Extract cldr-core dataset from CLDR database
2021-08-30 08:56:28 -04:00
set(CLDR_CORE_SOURCE cldr-core)
LibUnicode: Use consistent variable naming in unicode_data.cmake I kept mixing these up in my head.
2021-09-15 13:27:17 -04:00
set(CLDR_CORE_PATH "${CLDR_PATH}/${CLDR_CORE_SOURCE}")
LibUnicode: Extract cldr-core dataset from CLDR database
2021-08-30 08:56:28 -04:00
LibUnicode: Create a nearly empty generator for date-time formatting Similar to number formatting, the data for date-time formatting will be located in its own generated file. This extracts the cldr-dates package from the CLDR and sets up the generator plumbing to create the date-time data files.
2021-11-19 11:36:28 -05:00
set(CLDR_DATES_SOURCE cldr-dates-modern)
set(CLDR_DATES_PATH "${CLDR_PATH}/${CLDR_DATES_SOURCE}")
LibUnicode: Download Unicode's CLDR database and generate locale data The Unicode standard publishes a database known as the Common Locale Data Repository (CLDR). This is a massive set of data from which anyone implementing Unicode's Technical Standard #35 may generate their implementation: https://www.unicode.org/reports/tr35/ This commit updates LibUnicode to download the compressed database and extract a small subset. That subset is used to generate a list of available locales and the territories (AKA regions) associated with each locale.
2021-08-24 19:50:18 -04:00
set(CLDR_LOCALES_SOURCE cldr-localenames-modern)
LibUnicode: Use consistent variable naming in unicode_data.cmake I kept mixing these up in my head.
2021-09-15 13:27:17 -04:00
set(CLDR_LOCALES_PATH "${CLDR_PATH}/${CLDR_LOCALES_SOURCE}")
LibUnicode: Download Unicode's CLDR database and generate locale data The Unicode standard publishes a database known as the Common Locale Data Repository (CLDR). This is a massive set of data from which anyone implementing Unicode's Technical Standard #35 may generate their implementation: https://www.unicode.org/reports/tr35/ This commit updates LibUnicode to download the compressed database and extract a small subset. That subset is used to generate a list of available locales and the territories (AKA regions) associated with each locale.
2021-08-24 19:50:18 -04:00
LibUnicode: Extract cldr-misc dataset from CLDR database
2021-09-05 22:46:03 -04:00
set(CLDR_MISC_SOURCE cldr-misc-modern)
LibUnicode: Use consistent variable naming in unicode_data.cmake I kept mixing these up in my head.
2021-09-15 13:27:17 -04:00
set(CLDR_MISC_PATH "${CLDR_PATH}/${CLDR_MISC_SOURCE}")
LibUnicode: Extract cldr-misc dataset from CLDR database
2021-09-05 22:46:03 -04:00
LibUnicode: Extract cldr-numbers dataset from CLDR database This dataset holds the values needed to handle DisplayNames.prototype.of with a type option of "currency".
2021-08-26 08:31:31 -04:00
set(CLDR_NUMBERS_SOURCE cldr-numbers-modern)
LibUnicode: Use consistent variable naming in unicode_data.cmake I kept mixing these up in my head.
2021-09-15 13:27:17 -04:00
set(CLDR_NUMBERS_PATH "${CLDR_PATH}/${CLDR_NUMBERS_SOURCE}")
LibUnicode: Extract cldr-numbers dataset from CLDR database This dataset holds the values needed to handle DisplayNames.prototype.of with a type option of "currency".
2021-08-26 08:31:31 -04:00
LibUnicode: Parse and generate CLDR unit data for Intl.NumberFormat The units data is in another CLDR package, cldr-units.
2021-11-16 09:31:15 -05:00
set(CLDR_UNITS_SOURCE cldr-units-modern)
set(CLDR_UNITS_PATH "${CLDR_PATH}/${CLDR_UNITS_SOURCE}")
LibUnicode: Functionalize downloading of UCD data files
2021-09-15 08:46:56 -04:00
function(download_ucd_file url path)
if (NOT EXISTS "${path}")
get_filename_component(file "${path}" NAME)
message(STATUS "Downloading UCD ${file} from ${url}...")
file(DOWNLOAD "${url}" "${path}" INACTIVITY_TIMEOUT 10)
LibUnicode: Download and parse DerivedNormalizationProps UCD file This file contains the last properties that LibUnicode is not parsing. Much of the data in this file is not currently used; that is left as a FIXME for when String.prototype.normalize is implemented. Until then, only the code point properties are utilized for regular expression pattern escapes.
2021-08-10 15:29:28 -04:00
endif()
LibUnicode: Functionalize downloading of UCD data files
2021-09-15 08:46:56 -04:00
endfunction()
LibUnicode: Functionalize extraction of CLDR data files
2021-09-15 08:50:45 -04:00
function(extract_cldr_file source path)
if(EXISTS "${CLDR_ZIP_PATH}" AND NOT EXISTS "${path}")
message(STATUS "Extracting CLDR ${source} from ${CLDR_ZIP_PATH}...")
LibUnicode: Use consistent variable naming in unicode_data.cmake I kept mixing these up in my head.
2021-09-15 13:27:17 -04:00
execute_process(COMMAND unzip -q "${CLDR_ZIP_PATH}" "${source}/**" -d "${CLDR_PATH}" RESULT_VARIABLE unzip_result)
LibUnicode: Functionalize extraction of CLDR data files
2021-09-15 08:50:45 -04:00
if (NOT unzip_result EQUAL 0)
message(FATAL_ERROR "Failed to unzip ${source} from ${CLDR_ZIP_PATH} with status ${unzip_result}")
endif()
endif()
endfunction()
LibUnicode: Functionalize downloading of UCD data files
2021-09-15 08:46:56 -04:00
if (ENABLE_UNICODE_DATABASE_DOWNLOAD)
Meta: Move remove_unicode_data_if_version_changed to utils.cmake This function will be used by the time zone database parser. Move it to the common utilities file, and rename it remove_path_if_version_changed to be more generic.
2021-12-22 16:12:59 -05:00
remove_path_if_version_changed("${UCD_VERSION}" "${UCD_VERSION_FILE}" "${UCD_PATH}")
remove_path_if_version_changed("${CLDR_VERSION}" "${CLDR_VERSION_FILE}" "${CLDR_PATH}")
LibUnicode: Remove cached UCD and CLDR files when their version changes
2021-09-15 09:33:55 -04:00
LibUnicode: Functionalize downloading of UCD data files
2021-09-15 08:46:56 -04:00
download_ucd_file("${UNICODE_DATA_URL}" "${UNICODE_DATA_PATH}")
download_ucd_file("${SPECIAL_CASING_URL}" "${SPECIAL_CASING_PATH}")
download_ucd_file("${DERIVED_GENERAL_CATEGORY_URL}" "${DERIVED_GENERAL_CATEGORY_PATH}")
download_ucd_file("${PROP_LIST_URL}" "${PROP_LIST_PATH}")
download_ucd_file("${DERIVED_CORE_PROP_URL}" "${DERIVED_CORE_PROP_PATH}")
download_ucd_file("${DERIVED_BINARY_PROP_URL}" "${DERIVED_BINARY_PROP_PATH}")
download_ucd_file("${PROP_ALIAS_URL}" "${PROP_ALIAS_PATH}")
download_ucd_file("${PROP_VALUE_ALIAS_URL}" "${PROP_VALUE_ALIAS_PATH}")
Meta+LibUnicode: Provide code point names through library
2021-11-18 23:23:58 +01:00
download_ucd_file("${NAME_ALIAS_URL}" "${NAME_ALIAS_PATH}")
LibUnicode: Functionalize downloading of UCD data files
2021-09-15 08:46:56 -04:00
download_ucd_file("${SCRIPTS_URL}" "${SCRIPTS_PATH}")
download_ucd_file("${SCRIPT_EXTENSIONS_URL}" "${SCRIPT_EXTENSIONS_PATH}")
download_ucd_file("${EMOJI_DATA_URL}" "${EMOJI_DATA_PATH}")
download_ucd_file("${NORM_PROPS_URL}" "${NORM_PROPS_PATH}")
LibUnicode: Introduce a Unicode library for interacting with UCD files The Unicode standard publishes the Unicode Character Database (UCD) with information about every code point, such as each code point's upper case mapping. LibUnicode exists to download and parse UCD files at build time and to provide accessors to that data. As a start, LibUnicode includes upper- and lower-case code point converters.
2021-07-25 15:10:51 -04:00
LibUnicode: Functionalize extraction of CLDR data files
2021-09-15 08:50:45 -04:00
if (NOT EXISTS "${CLDR_ZIP_PATH}")
LibUnicode: Download Unicode's CLDR database and generate locale data The Unicode standard publishes a database known as the Common Locale Data Repository (CLDR). This is a massive set of data from which anyone implementing Unicode's Technical Standard #35 may generate their implementation: https://www.unicode.org/reports/tr35/ This commit updates LibUnicode to download the compressed database and extract a small subset. That subset is used to generate a list of available locales and the territories (AKA regions) associated with each locale.
2021-08-24 19:50:18 -04:00
message(STATUS "Downloading CLDR database from ${CLDR_ZIP_URL}...")
LibUnicode: Functionalize extraction of CLDR data files
2021-09-15 08:50:45 -04:00
file(DOWNLOAD "${CLDR_ZIP_URL}" "${CLDR_ZIP_PATH}" INACTIVITY_TIMEOUT 10)
LibUnicode: Download Unicode's CLDR database and generate locale data The Unicode standard publishes a database known as the Common Locale Data Repository (CLDR). This is a massive set of data from which anyone implementing Unicode's Technical Standard #35 may generate their implementation: https://www.unicode.org/reports/tr35/ This commit updates LibUnicode to download the compressed database and extract a small subset. That subset is used to generate a list of available locales and the territories (AKA regions) associated with each locale.
2021-08-24 19:50:18 -04:00
endif()
LibUnicode: Functionalize extraction of CLDR data files
2021-09-15 08:50:45 -04:00
extract_cldr_file("${CLDR_CORE_SOURCE}" "${CLDR_CORE_PATH}")
LibUnicode: Create a nearly empty generator for date-time formatting Similar to number formatting, the data for date-time formatting will be located in its own generated file. This extracts the cldr-dates package from the CLDR and sets up the generator plumbing to create the date-time data files.
2021-11-19 11:36:28 -05:00
extract_cldr_file("${CLDR_DATES_SOURCE}" "${CLDR_DATES_PATH}")
LibUnicode: Functionalize extraction of CLDR data files
2021-09-15 08:50:45 -04:00
extract_cldr_file("${CLDR_LOCALES_SOURCE}" "${CLDR_LOCALES_PATH}")
extract_cldr_file("${CLDR_MISC_SOURCE}" "${CLDR_MISC_PATH}")
extract_cldr_file("${CLDR_NUMBERS_SOURCE}" "${CLDR_NUMBERS_PATH}")
LibUnicode: Parse and generate CLDR unit data for Intl.NumberFormat The units data is in another CLDR package, cldr-units.
2021-11-16 09:31:15 -05:00
extract_cldr_file("${CLDR_UNITS_SOURCE}" "${CLDR_UNITS_PATH}")
LibUnicode: Functionalize extraction of CLDR data files
2021-09-15 08:50:45 -04:00
LibUnicode: Make unicode data generation logic more relocatable The previous logic had several checks for Lagom directories and subdirectories. All we really want to do for these header checks is make sure that the files end up in an included folder prefixed with LibUnicode. We also don't need to hard code the path to the generator, the $<TARGET_FILES> generator expression can create the path for us.
2021-07-29 02:18:52 -06:00
set(UNICODE_DATA_HEADER LibUnicode/UnicodeData.h)
set(UNICODE_DATA_IMPLEMENTATION LibUnicode/UnicodeData.cpp)
LibUnicode: Introduce a Unicode library for interacting with UCD files The Unicode standard publishes the Unicode Character Database (UCD) with information about every code point, such as each code point's upper case mapping. LibUnicode exists to download and parse UCD files at build time and to provide accessors to that data. As a start, LibUnicode includes upper- and lower-case code point converters.
2021-07-25 15:10:51 -04:00
LibUnicode: Create a nearly empty generator for date-time formatting Similar to number formatting, the data for date-time formatting will be located in its own generated file. This extracts the cldr-dates package from the CLDR and sets up the generator plumbing to create the date-time data files.
2021-11-19 11:36:28 -05:00
set(UNICODE_DATE_TIME_FORMAT_HEADER LibUnicode/UnicodeDateTimeFormat.h)
set(UNICODE_DATE_TIME_FORMAT_IMPLEMENTATION LibUnicode/UnicodeDateTimeFormat.cpp)
LibUnicode: Download Unicode's CLDR database and generate locale data The Unicode standard publishes a database known as the Common Locale Data Repository (CLDR). This is a massive set of data from which anyone implementing Unicode's Technical Standard #35 may generate their implementation: https://www.unicode.org/reports/tr35/ This commit updates LibUnicode to download the compressed database and extract a small subset. That subset is used to generate a list of available locales and the territories (AKA regions) associated with each locale.
2021-08-24 19:50:18 -04:00
set(UNICODE_LOCALE_HEADER LibUnicode/UnicodeLocale.h)
set(UNICODE_LOCALE_IMPLEMENTATION LibUnicode/UnicodeLocale.cpp)
LibUnicode: Move number formatting code generator to UnicodeNumberFormat
2021-11-12 09:09:48 -05:00
set(UNICODE_NUMBER_FORMAT_HEADER LibUnicode/UnicodeNumberFormat.h)
set(UNICODE_NUMBER_FORMAT_IMPLEMENTATION LibUnicode/UnicodeNumberFormat.cpp)
CMake: Add custom target to build only the generated sources This is needed so all headers and files exist on disk, so that the sonar cloud analyzer can find them when executing the compilation commands contained in compile_commands.json, without actually building. Co-authored-by: Andrew Kaster <akaster@serenityos.org>
2021-08-29 19:27:50 -07:00
set(UNICODE_META_TARGET_PREFIX LibUnicode_)
LibUnicode: Make unicode data generation logic more relocatable The previous logic had several checks for Lagom directories and subdirectories. All we really want to do for these header checks is make sure that the files end up in an included folder prefixed with LibUnicode. We also don't need to hard code the path to the generator, the $<TARGET_FILES> generator expression can create the path for us.
2021-07-29 02:18:52 -06:00
if (CMAKE_CURRENT_BINARY_DIR MATCHES ".*/LibUnicode") # Serenity build.
set(UNICODE_DATA_HEADER UnicodeData.h)
set(UNICODE_DATA_IMPLEMENTATION UnicodeData.cpp)
LibUnicode: Download Unicode's CLDR database and generate locale data The Unicode standard publishes a database known as the Common Locale Data Repository (CLDR). This is a massive set of data from which anyone implementing Unicode's Technical Standard #35 may generate their implementation: https://www.unicode.org/reports/tr35/ This commit updates LibUnicode to download the compressed database and extract a small subset. That subset is used to generate a list of available locales and the territories (AKA regions) associated with each locale.
2021-08-24 19:50:18 -04:00
LibUnicode: Create a nearly empty generator for date-time formatting Similar to number formatting, the data for date-time formatting will be located in its own generated file. This extracts the cldr-dates package from the CLDR and sets up the generator plumbing to create the date-time data files.
2021-11-19 11:36:28 -05:00
set(UNICODE_DATE_TIME_FORMAT_HEADER UnicodeDateTimeFormat.h)
set(UNICODE_DATE_TIME_FORMAT_IMPLEMENTATION UnicodeDateTimeFormat.cpp)
LibUnicode: Download Unicode's CLDR database and generate locale data The Unicode standard publishes a database known as the Common Locale Data Repository (CLDR). This is a massive set of data from which anyone implementing Unicode's Technical Standard #35 may generate their implementation: https://www.unicode.org/reports/tr35/ This commit updates LibUnicode to download the compressed database and extract a small subset. That subset is used to generate a list of available locales and the territories (AKA regions) associated with each locale.
2021-08-24 19:50:18 -04:00
set(UNICODE_LOCALE_HEADER UnicodeLocale.h)
set(UNICODE_LOCALE_IMPLEMENTATION UnicodeLocale.cpp)
LibUnicode: Move number formatting code generator to UnicodeNumberFormat
2021-11-12 09:09:48 -05:00
set(UNICODE_NUMBER_FORMAT_HEADER UnicodeNumberFormat.h)
set(UNICODE_NUMBER_FORMAT_IMPLEMENTATION UnicodeNumberFormat.cpp)
CMake: Add custom target to build only the generated sources This is needed so all headers and files exist on disk, so that the sonar cloud analyzer can find them when executing the compilation commands contained in compile_commands.json, without actually building. Co-authored-by: Andrew Kaster <akaster@serenityos.org>
2021-08-29 19:27:50 -07:00
set(UNICODE_META_TARGET_PREFIX "")
Meta: Add LibUnicode (and its tests) to Lagom This is primarily to allow using LibUnicode within LibJS and its REPL. Note: this seems to be the first time that a Lagom dependency requires generated source files. For this to work, some of Lagom's CMakeLists.txt commands needed to be re-organized to include the CMake files that fetch and parse UnicodeData.txt. The paths required to invoke the generator also differ depending on what is currently building (SerenityOS vs. Lagom as part of the Serenity build vs. a standalone Lagom build).
2021-07-25 15:15:47 -04:00
endif()
LibUnicode: Functionalize the Unicode generator CMake commands Makes it a bit easier to add a new generator.
2021-11-19 11:25:02 -05:00
invoke_generator(
"UnicodeData"
Lagom::GenerateUnicodeData
Meta: Move invoke_generator to utils.cmake
2021-12-22 16:30:21 -05:00
"${UCD_VERSION_FILE}"
"${UNICODE_META_TARGET_PREFIX}"
LibUnicode: Functionalize the Unicode generator CMake commands Makes it a bit easier to add a new generator.
2021-11-19 11:25:02 -05:00
"${UNICODE_DATA_HEADER}"
"${UNICODE_DATA_IMPLEMENTATION}"
arguments -u "${UNICODE_DATA_PATH}" -s "${SPECIAL_CASING_PATH}" -g "${DERIVED_GENERAL_CATEGORY_PATH}" -p "${PROP_LIST_PATH}" -d "${DERIVED_CORE_PROP_PATH}" -b "${DERIVED_BINARY_PROP_PATH}" -a "${PROP_ALIAS_PATH}" -v "${PROP_VALUE_ALIAS_PATH}" -r "${SCRIPTS_PATH}" -x "${SCRIPT_EXTENSIONS_PATH}" -e "${EMOJI_DATA_PATH}" -m "${NAME_ALIAS_PATH}" -n "${NORM_PROPS_PATH}"
LibUnicode: Introduce a Unicode library for interacting with UCD files The Unicode standard publishes the Unicode Character Database (UCD) with information about every code point, such as each code point's upper case mapping. LibUnicode exists to download and parse UCD files at build time and to provide accessors to that data. As a start, LibUnicode includes upper- and lower-case code point converters.
2021-07-25 15:10:51 -04:00
)
LibUnicode: Create a nearly empty generator for date-time formatting Similar to number formatting, the data for date-time formatting will be located in its own generated file. This extracts the cldr-dates package from the CLDR and sets up the generator plumbing to create the date-time data files.
2021-11-19 11:36:28 -05:00
invoke_generator(
"UnicodeDateTimeFormat"
Lagom::GenerateUnicodeDateTimeFormat
Meta: Move invoke_generator to utils.cmake
2021-12-22 16:30:21 -05:00
"${CLDR_VERSION_FILE}"
"${UNICODE_META_TARGET_PREFIX}"
LibUnicode: Create a nearly empty generator for date-time formatting Similar to number formatting, the data for date-time formatting will be located in its own generated file. This extracts the cldr-dates package from the CLDR and sets up the generator plumbing to create the date-time data files.
2021-11-19 11:36:28 -05:00
"${UNICODE_DATE_TIME_FORMAT_HEADER}"
"${UNICODE_DATE_TIME_FORMAT_IMPLEMENTATION}"
LibUnicode: Parse and generate regional hour cycles Unlike most data in the CLDR, hour cycles are not stored on a per-locale basis. Instead, they are keyed by a string that is usually a region, but sometimes is a locale. Therefore, given a locale, to determine the hour cycles for that locale, we: 1. Check if the locale itself is assigned hour cycles. 2. If the locale has a region, check if that region is assigned hour cycles. 3. Otherwise, maximize that locale, and if the maximized locale has a region, check if that region is assigned hour cycles. 4. If the above all fail, fallback to the "001" region. Further, each locale's default hour cycle is the first assigned hour cycle.
2021-11-27 20:57:21 -05:00
arguments -r "${CLDR_CORE_PATH}" -d "${CLDR_DATES_PATH}"
LibUnicode: Create a nearly empty generator for date-time formatting Similar to number formatting, the data for date-time formatting will be located in its own generated file. This extracts the cldr-dates package from the CLDR and sets up the generator plumbing to create the date-time data files.
2021-11-19 11:36:28 -05:00
)
LibUnicode: Functionalize the Unicode generator CMake commands Makes it a bit easier to add a new generator.
2021-11-19 11:25:02 -05:00
invoke_generator(
"UnicodeLocale"
Lagom::GenerateUnicodeLocale
Meta: Move invoke_generator to utils.cmake
2021-12-22 16:30:21 -05:00
"${CLDR_VERSION_FILE}"
"${UNICODE_META_TARGET_PREFIX}"
LibUnicode: Functionalize the Unicode generator CMake commands Makes it a bit easier to add a new generator.
2021-11-19 11:25:02 -05:00
"${UNICODE_LOCALE_HEADER}"
"${UNICODE_LOCALE_IMPLEMENTATION}"
LibUnicode: Parse and generate calendar (ca) Unicode keywords Also removes a few fly-by "StringView x = nullptr;" unnecessary initializers.
2021-11-28 10:39:55 -05:00
arguments -r "${CLDR_CORE_PATH}" -l "${CLDR_LOCALES_PATH}" -m "${CLDR_MISC_PATH}" -n "${CLDR_NUMBERS_PATH}" -d "${CLDR_DATES_PATH}"
LibUnicode: Download Unicode's CLDR database and generate locale data The Unicode standard publishes a database known as the Common Locale Data Repository (CLDR). This is a massive set of data from which anyone implementing Unicode's Technical Standard #35 may generate their implementation: https://www.unicode.org/reports/tr35/ This commit updates LibUnicode to download the compressed database and extract a small subset. That subset is used to generate a list of available locales and the territories (AKA regions) associated with each locale.
2021-08-24 19:50:18 -04:00
)
LibUnicode: Functionalize the Unicode generator CMake commands Makes it a bit easier to add a new generator.
2021-11-19 11:25:02 -05:00
invoke_generator(
"UnicodeNumberFormat"
Lagom::GenerateUnicodeNumberFormat
Meta: Move invoke_generator to utils.cmake
2021-12-22 16:30:21 -05:00
"${CLDR_VERSION_FILE}"
"${UNICODE_META_TARGET_PREFIX}"
LibUnicode: Functionalize the Unicode generator CMake commands Makes it a bit easier to add a new generator.
2021-11-19 11:25:02 -05:00
"${UNICODE_NUMBER_FORMAT_HEADER}"
"${UNICODE_NUMBER_FORMAT_IMPLEMENTATION}"
arguments -n "${CLDR_NUMBERS_PATH}" -u "${CLDR_UNITS_PATH}"
LibUnicode: Move number formatting code generator to UnicodeNumberFormat
2021-11-12 09:09:48 -05:00
)
set(UNICODE_DATA_SOURCES
${UNICODE_DATA_HEADER}
${UNICODE_DATA_IMPLEMENTATION}
LibUnicode: Create a nearly empty generator for date-time formatting Similar to number formatting, the data for date-time formatting will be located in its own generated file. This extracts the cldr-dates package from the CLDR and sets up the generator plumbing to create the date-time data files.
2021-11-19 11:36:28 -05:00
${UNICODE_DATE_TIME_FORMAT_HEADER}
${UNICODE_DATE_TIME_FORMAT_IMPLEMENTATION}
LibUnicode: Move number formatting code generator to UnicodeNumberFormat
2021-11-12 09:09:48 -05:00
${UNICODE_LOCALE_HEADER}
${UNICODE_LOCALE_IMPLEMENTATION}
${UNICODE_NUMBER_FORMAT_HEADER}
${UNICODE_NUMBER_FORMAT_IMPLEMENTATION}
)
LibUnicode: Introduce a Unicode library for interacting with UCD files The Unicode standard publishes the Unicode Character Database (UCD) with information about every code point, such as each code point's upper case mapping. LibUnicode exists to download and parse UCD files at build time and to provide accessors to that data. As a start, LibUnicode includes upper- and lower-case code point converters.
2021-07-25 15:10:51 -04:00
endif()
Reference in a new issue Copy permalink