| Property | \nMeaning | \nUsage | \n
|---|---|---|
table\n | \n The BigQuery table in the format [[project:]dataset.]table.\n It is recommended to use the path parameter of\n load()/save() instead. This option has been\n deprecated and will be removed in a future version.\n (Deprecated)\n | \n Read/Write | \n
dataset\n | \n The dataset containing the table. This option should be used with\n standard table and views, but not when loading query results.\n (Optional unless omitted in table)\n | \n Read/Write | \n
project\n | \n The Google Cloud Project ID of the table. This option should be used with\n standard table and views, but not when loading query results.\n (Optional. Defaults to the project of the Service Account being used)\n | \n Read/Write | \n
billingProject\n | \n The Google Cloud Project ID to use for billing (API calls, query execution).\n (Optional. Defaults to the project of the Service Account being used)\n | \n Read/Write | \n
parentProject\n | \n (Deprecated) Alias for billingProject.\n (Optional. Defaults to the project of the Service Account being used)\n | \n Read/Write | \n
location\n | \n The BigQuery location where the data resides (e.g. US, EU, asia-northeast1).\n (Optional. Defaults to BigQuery default)\n | \n Read/Write | \n
maxParallelism\n | \n The maximal number of partitions to split the data into. Actual number\n may be less if BigQuery deems the data small enough. If there are not\n enough executors to schedule a reader per partition, some partitions may\n be empty.\n Important: The old parameter ( parallelism) is\n still supported but in deprecated mode. It will ve removed in\n version 1.0 of the connector.\n (Optional. Defaults to the larger of the preferredMinParallelism and 20,000).)\n | \n Read | \n
preferredMinParallelism\n | \n The preferred minimal number of partitions to split the data into. Actual number\n may be less if BigQuery deems the data small enough. If there are not\n enough executors to schedule a reader per partition, some partitions may\n be empty.\n (Optional. Defaults to the smallest of 3 times the application's default parallelism\n and maxParallelism.)\n | \n Read | \n
viewsEnabled\n | \n Enables the connector to read from views and not only tables. Please read\n the relevant section before activating\n this option.\n (Optional. Defaults to false)\n | \n Read | \n
readDataFormat\n | \n Data Format for reading from BigQuery. Options : ARROW, AVRO\n (Optional. Defaults to ARROW)\n | \n Read | \n
optimizedEmptyProjection\n | \n The connector uses an optimized empty projection (select without any\n columns) logic, used for count() execution. This logic takes\n the data directly from the table metadata or performs a much efficient\n `SELECT COUNT(*) WHERE...` in case there is a filter. You can cancel the\n use of this logic by setting this option to false.\n (Optional, defaults to true)\n | \n Read | \n
pushAllFilters\n | \n If set to true, the connector pushes all the filters Spark can delegate\n to BigQuery Storage API. This reduces amount of data that needs to be sent from\n BigQuery Storage API servers to Spark clients. This option has been\n deprecated and will be removed in a future version.\n (Optional, defaults to true)\n (Deprecated)\n | \n Read | \n
bigQueryJobLabel\n | \n Can be used to add labels to the connector initiated query and load\n BigQuery jobs. Multiple labels can be set.\n (Optional)\n | \n Read | \n
bigQueryTableLabel\n | \n Can be used to add labels to the table while writing to a table. Multiple\n labels can be set.\n (Optional)\n | \n Write | \n
traceApplicationName\n | \n Application name used to trace BigQuery Storage read and write sessions.\n Setting the application name is required to set the trace ID on the\n sessions.\n (Optional)\n | \n Read | \n
traceJobId\n | \n Job ID used to trace BigQuery Storage read and write sessions.\n (Optional, defaults to the Dataproc job ID is exists, otherwise uses\n the Spark application ID)\n | \n Read | \n
createDisposition\n | \n Specifies whether the job is allowed to create new tables. The permitted\n values are:\n
(Optional. Default to CREATE_IF_NEEDED).\n | \n Write | \n
writeMethod\n | \n Controls the method\n in which the data is written to BigQuery. Available values are direct\n to use the BigQuery Storage Write API and indirect which writes the\n data first to GCS and then triggers a BigQuery load operation. See more\n here\n (Optional, defaults to indirect)\n | \n Write | \n
writeAtLeastOnce\n | \n Guarantees that data is written to BigQuery at least once. This is a lesser\n guarantee than exactly once. This is suitable for streaming scenarios\n in which data is continuously being written in small batches.\n (Optional. Defaults to false)\n Supported only by the `DIRECT` write method and mode is NOT `Overwrite`.\n | \n Write | \n
temporaryGcsBucket\n | \n The GCS bucket that temporarily holds the data before it is loaded to\n BigQuery. Required unless set in the Spark configuration\n (spark.conf.set(...)).\n Defaults to the `fs.gs.system.bucket` if exists, for example on Google Cloud Dataproc clusters, starting version 0.42.0.\n Supported only by the `INDIRECT` write method.\n | \n Write | \n
persistentGcsBucket\n | \n The GCS bucket that holds the data before it is loaded to\n BigQuery. If informed, the data won't be deleted after write data\n into BigQuery.\n Supported only by the `INDIRECT` write method.\n | \n Write | \n
persistentGcsPath\n | \n The GCS path that holds the data before it is loaded to\n BigQuery. Used only with persistentGcsBucket.\n Not supported by the `DIRECT` write method.\n | \n Write | \n
intermediateFormat\n | \n The format of the data before it is loaded to BigQuery, values can be\n either \"parquet\",\"orc\" or \"avro\". In order to use the Avro format, the\n spark-avro package must be added in runtime.\n (Optional. Defaults to parquet). On write only. Supported only for the `INDIRECT` write method.\n | \n Write | \n
useAvroLogicalTypes\n | \n When loading from Avro (`.option(\"intermediateFormat\", \"avro\")`), BigQuery uses the underlying Avro types instead of the logical types [by default](https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-avro#logical_types). Supplying this option converts Avro logical types to their corresponding BigQuery data types.\n (Optional. Defaults to false). On write only.\n | \n Write | \n
datePartition\n | \n The date partition the data is going to be written to. Should be a date string\n given in the format YYYYMMDD. Can be used to overwrite the data of\n\t a single partition, like this: (Optional). On write only.\n Can also be used with different partition types like:\n HOUR: YYYYMMDDHH\n MONTH: YYYYMM\n YEAR: YYYY\n Not supported by the `DIRECT` write method.\n | \n Write | \n
partitionField\n | \n If this field is specified, the table is partitioned by this field.\n For Time partitioning, specify together with the option `partitionType`.\n For Integer-range partitioning, specify together with the 3 options: `partitionRangeStart`, `partitionRangeEnd, `partitionRangeInterval`.\n The field must be a top-level TIMESTAMP or DATE field for Time partitioning, or INT64 for Integer-range partitioning. Its mode must be NULLABLE\n or REQUIRED.\n If the option is not set for a Time partitioned table, then the table will be partitioned by pseudo\n column, referenced via either '_PARTITIONTIME' as TIMESTAMP type, or\n '_PARTITIONDATE' as DATE type.\n (Optional).\n Not supported by the `DIRECT` write method.\n | \n Write | \n
partitionExpirationMs\n | \n Number of milliseconds for which to keep the storage for partitions in the table.\n The storage in a partition will have an expiration time of its partition time plus this value.\n (Optional).\n Not supported by the `DIRECT` write method.\n | \n Write | \n
partitionType\n | \n Used to specify Time partitioning.\n Supported types are: HOUR, DAY, MONTH, YEAR\n This option is mandatory for a target table to be Time partitioned.\n (Optional. Defaults to DAY if PartitionField is specified).\n Not supported by the `DIRECT` write method.\n | \n Write | \n
partitionRangeStart,\n partitionRangeEnd,\n partitionRangeInterval\n | \n Used to specify Integer-range partitioning.\n These options are mandatory for a target table to be Integer-range partitioned.\n All 3 options must be specified.\n Not supported by the `DIRECT` write method.\n | \n Write | \n
clusteredFields\n | \n A string of non-repeated, top level columns seperated by comma.\n (Optional).\n | \n Write | \n
allowFieldAddition\n | \n Adds the ALLOW_FIELD_ADDITION\n SchemaUpdateOption to the BigQuery LoadJob. Allowed values are true and false.\n (Optional. Default to false).\n Supported only by the `INDIRECT` write method.\n | \n Write | \n
allowFieldRelaxation\n | \n Adds the ALLOW_FIELD_RELAXATION\n SchemaUpdateOption to the BigQuery LoadJob. Allowed values are true and false.\n (Optional. Default to false).\n Supported only by the `INDIRECT` write method.\n | \n Write | \n
proxyAddress\n | \n Address of the proxy server. The proxy must be a HTTP proxy and address should be in the `host:port` format.\n Can be alternatively set in the Spark configuration (spark.conf.set(...)) or in Hadoop\n Configuration (fs.gs.proxy.address).\n (Optional. Required only if connecting to GCP via proxy.)\n | \n Read/Write | \n
proxyUsername\n | \n The userName used to connect to the proxy. Can be alternatively set in the Spark configuration\n (spark.conf.set(...)) or in Hadoop Configuration (fs.gs.proxy.username).\n (Optional. Required only if connecting to GCP via proxy with authentication.)\n | \n Read/Write | \n
proxyPassword\n | \n The password used to connect to the proxy. Can be alternatively set in the Spark configuration\n (spark.conf.set(...)) or in Hadoop Configuration (fs.gs.proxy.password).\n (Optional. Required only if connecting to GCP via proxy with authentication.)\n | \n Read/Write | \n
httpMaxRetry\n | \n The maximum number of retries for the low-level HTTP requests to BigQuery. Can be alternatively set in the\n Spark configuration (spark.conf.set(\"httpMaxRetry\", ...)) or in Hadoop Configuration\n (fs.gs.http.max.retry).\n (Optional. Default is 10)\n | \n Read/Write | \n
httpConnectTimeout\n | \n The timeout in milliseconds to establish a connection with BigQuery. Can be alternatively set in the\n Spark configuration (spark.conf.set(\"httpConnectTimeout\", ...)) or in Hadoop Configuration\n (fs.gs.http.connect-timeout).\n (Optional. Default is 60000 ms. 0 for an infinite timeout, a negative number for 20000)\n | \n Read/Write | \n
httpReadTimeout\n | \n The timeout in milliseconds to read data from an established connection. Can be alternatively set in the\n Spark configuration (spark.conf.set(\"httpReadTimeout\", ...)) or in Hadoop Configuration\n (fs.gs.http.read-timeout).\n (Optional. Default is 60000 ms. 0 for an infinite timeout, a negative number for 20000)\n | \n Read | \n
arrowCompressionCodec\n | \n Compression codec while reading from a BigQuery table when using Arrow format. Options :\n ZSTD (Zstandard compression),\n LZ4_FRAME (https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md),\n COMPRESSION_UNSPECIFIED. The recommended compression codec is ZSTD\n while using Java.\n (Optional. Defaults to COMPRESSION_UNSPECIFIED which means no compression will be used)\n | \n Read | \n
responseCompressionCodec\n | \n Compression codec used to compress the ReadRowsResponse data. Options:\n RESPONSE_COMPRESSION_CODEC_UNSPECIFIED,\n RESPONSE_COMPRESSION_CODEC_LZ4\n (Optional. Defaults to RESPONSE_COMPRESSION_CODEC_UNSPECIFIED which means no compression will be used)\n | \n Read | \n
cacheExpirationTimeInMinutes\n | \n The expiration time of the in-memory cache storing query information.\n To disable caching, set the value to 0.\n (Optional. Defaults to 15 minutes)\n | \n Read | \n
enableModeCheckForSchemaFields\n | \n Checks the mode of every field in destination schema to be equal to the mode in corresponding source field schema, during DIRECT write.\n Default value is true i.e., the check is done by default. If set to false the mode check is ignored.\n | \n Write | \nenableListInference\n | \n Indicates whether to use schema inference specifically when the mode is Parquet (https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#parquetoptions).\n Defaults to false.\n \n | \n Write | \n \n
bqChannelPoolSize | \n The (fixed) size of the gRPC channel pool created by the BigQueryReadClient.\n For optimal performance, this should be set to at least the number of cores on the cluster executors.\n | \n Read | \n
createReadSessionTimeoutInSeconds\n | \n The timeout in seconds to create a ReadSession when reading a table.\n For Extremely large table this value should be increased.\n (Optional. Defaults to 600 seconds)\n | \n Read | \n
queryJobPriority\n | \n Priority levels set for the job while reading data from BigQuery query. The permitted values are:\n
(Optional. Defaults to INTERACTIVE)\n | \n Read/Write | \n
destinationTableKmsKeyName\n | \n Describes the Cloud KMS encryption key that will be used to protect destination BigQuery\n table. The BigQuery Service Account associated with your project requires access to this\n encryption key. for further Information about using CMEK with BigQuery see\n [here](https://cloud.google.com/bigquery/docs/customer-managed-encryption#key_resource_id).\n Notice: The table will be encrypted by the key only if it created by the\n connector. A pre-existing unencrypted table won't be encrypted just by setting this option.\n (Optional)\n | \n Write | \n
allowMapTypeConversion\n | \n Boolean config to disable conversion from BigQuery records to Spark MapType\n when the record has two subfields with field names as key and value.\n Default value is true which allows the conversion.\n (Optional)\n | \n Read | \n
spark.sql.sources.partitionOverwriteMode\n | \n Config to specify the overwrite mode on write when the table is range/time partitioned.\n Currently supportd two modes : STATIC and DYNAMIC. In STATIC mode,\n the entire table is overwritten. In DYNAMIC mode, the data is overwritten by partitions of the existing table.\n The default value is STATIC.\n (Optional)\n | \n Write | \n
enableReadSessionCaching\n | \n Boolean config to disable read session caching. Caches BigQuery read sessions to allow for faster Spark query planning.\n Default value is true.\n (Optional)\n | \n Read | \n
readSessionCacheDurationMins\n | \n Config to set the read session caching duration in minutes. Only works if enableReadSessionCaching is true (default).\n Allows specifying the duration to cache read sessions for. Maximum allowed value is 300.\n Default value is 5.\n (Optional)\n | \n Read | \n
bigQueryJobTimeoutInMinutes\n | \n Config to set the BigQuery job timeout in minutes.\n Default value is 360 minutes.\n (Optional)\n | \n Read/Write | \n
snapshotTimeMillis\n | \n A timestamp specified in milliseconds to use to read a table snapshot.\n By default this is not set and the latest version of a table is read.\n (Optional)\n | \n Read | \n
bigNumericDefaultPrecision\n | \n An alternative default precision for BigNumeric fields, as the BigQuery default is too wide for Spark. Values can be between 1 and 38.\n This default is used only when the field has an unparameterized BigNumeric type.\n Please note that there might be data loss if the actual data's precision is more than what is specified.\n (Optional)\n | \n Read/Write | \n
bigNumericDefaultScale\n | \n An alternative default scale for BigNumeric fields. Values can be between 0 and 38, and less than bigNumericFieldsPrecision.\n This default is used only when the field has an unparameterized BigNumeric type.\n Please note that there might be data loss if the actual data's scale is more than what is specified.\n (Optional)\n | \n Read/Write | \n
credentialsScopes\n | \n Replaces the scopes of the Google Credentials if the credentials type supports that.\n If scope replacement is not supported then it does nothing.\n The value should be a comma separated list of valid scopes.\n (Optional)\n | \n Read/Write | \n
| BigQuery Standard SQL Data Type \n | \nSpark SQL\n \nData Type\n | \n Notes\n | \n
BOOL\n | \n BooleanType\n | \n \n | \n
INT64\n | \n LongType\n | \n \n | \n
FLOAT64\n | \n DoubleType\n | \n \n | \n
NUMERIC\n | \n DecimalType\n | \n \n Please refer to Numeric and BigNumeric support\n | \n
BIGNUMERIC\n | \n DecimalType\n | \n \n Please refer to Numeric and BigNumeric support\n | \n
STRING\n | \n StringType\n | \n \n | \n
BYTES\n | \n BinaryType\n | \n \n | \n
STRUCT\n | \n StructType\n | \n \n | \n
ARRAY\n | \n ArrayType\n | \n \n | \n
TIMESTAMP\n | \n TimestampType\n | \n \n | \n
DATE\n | \n DateType\n | \n \n | \n
DATETIME\n | \n StringType, TimestampNTZType*\n | \n Spark has no DATETIME type.\n \n Spark string can be written to an existing BQ DATETIME column provided it is in the format for BQ DATETIME literals.\n \n * For Spark 3.4+, BQ DATETIME is read as Spark's TimestampNTZ type i.e. java LocalDateTime\n | \n
TIME\n | \n LongType, StringType*\n | \n Spark has no TIME type. The generated longs, which indicate microseconds since midnight can be safely cast to TimestampType, but this causes the date to be inferred as the current day. Thus times are left as longs and user can cast if they like.\n \nWhen casting to Timestamp TIME have the same TimeZone issues as DATETIME\n \n* Spark string can be written to an existing BQ TIME column provided it is in the format for BQ TIME literals.\n | \n
JSON\n | \n StringType\n | \n Spark has no JSON type. The values are read as String. In order to write JSON back to BigQuery, the following conditions are REQUIRED:\n
| \n
ARRAY<STRUCT<key,value>>\n | \n MapType\n | \n BigQuery has no MAP type, therefore similar to other conversions like Apache Avro and BigQuery Load jobs, the connector converts a Spark Map to a REPEATED STRUCT<key,value>.\n This means that while writing and reading of maps is available, running a SQL on BigQuery that uses map semantics is not supported.\n To refer to the map's values using BigQuery SQL, please check the BigQuery documentation.\n Due to these incompatibilities, a few restrictions apply:\n
| \n
[maxParallelism](#properties) and [preferredMinParallelism](#properties) can be configured explicitly to control the number of partitions.\n\n## Tagging BigQuery Resources\n\nIn order to support tracking the usage of BigQuery resources the connectors\noffers the following options to tag BigQuery resources:\n\n### Adding BigQuery Jobs Labels\n\nThe connector can launch BigQuery load and query jobs. Adding labels to the jobs\nis done in the following manner:\n```\nspark.conf.set(\"bigQueryJobLabel.cost_center\", \"analytics\")\nspark.conf.set(\"bigQueryJobLabel.usage\", \"nightly_etl\")\n```\nThis will create labels `cost_center`=`analytics` and `usage`=`nightly_etl`.\n\n### Adding BigQuery Storage Trace ID\n\nUsed to annotate the read and write sessions. The trace ID is of the format\n`Spark:ApplicationName:JobID`. This is an opt-in option, and to use it the user\nneed to set the `traceApplicationName` property. JobID is auto generated by the\nDataproc job ID, with a fallback to the Spark application ID (such as\n`application_1648082975639_0001`). The Job ID can be overridden by setting the\n`traceJobId` option. Notice that the total length of the trace ID cannot be over\n256 characters.\n\n## Using in Jupyter Notebooks\n\nThe connector can be used in [Jupyter notebooks](https://jupyter.org/) even if\nit is not installed on the Spark cluster. It can be added as an external jar in\nusing the following code:\n\n**Python:**\n```python\nfrom pyspark.sql import SparkSession\nspark = SparkSession.builder \\\n .config(\"spark.jars.packages\", \"com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:${next-release-tag}\") \\\n .getOrCreate()\ndf = spark.read.format(\"bigquery\") \\\n .load(\"dataset.table\")\n```\n\n**Scala:**\n```scala\nval spark = SparkSession.builder\n.config(\"spark.jars.packages\", \"com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:${next-release-tag}\")\n.getOrCreate()\nval df = spark.read.format(\"bigquery\")\n.load(\"dataset.table\")\n```\n\nIn case Spark cluster is using Scala 2.12 (it's optional for Spark 2.4.x,\nmandatory in 3.0.x), then the relevant package is\ncom.google.cloud.spark:spark-bigquery-with-dependencies_**2.12**:${next-release-tag}. In\norder to know which Scala version is used, please run the following code:\n\n**Python:**\n```python\nspark.sparkContext._jvm.scala.util.Properties.versionString()\n```\n\n**Scala:**\n```python\nscala.util.Properties.versionString\n```\n## Compiling against the connector\n\nUnless you wish to use the implicit Scala API `spark.read.bigquery(\"TABLE_ID\")`, there is no need to compile against the connector.\n\nTo include the connector in your project:\n\n### Maven\n\n```xml\n| Metric Name | \nDescription | \n
|---|---|
bytes read | \n number of BigQuery bytes read | \n
rows read | \n number of BigQuery rows read | \n
scan time | \n the amount of time spent between read rows response requested to obtained across all the executors, in milliseconds. | \n
parse time | \n the amount of time spent for parsing the rows read across all the executors, in milliseconds. | \n
spark time | \n the amount of time spent in spark to process the queries (i.e., apart from scanning and parsing), across all the executors, in milliseconds. | \n
coalesce on the DataFrame to mitigate this problem.\n\n```\ndesiredPartitionCount = 5\ndfNew = df.coalesce(desiredPartitionCount)\ndfNew.write\n```\n\nA rule of thumb is to have a single partition handle at least 1GB of data.\n\nAlso note that a job running with the `writeAtLeastOnce` property turned on will not encounter CreateWriteStream\nquota errors.\n\n### How do I authenticate outside GCE / Dataproc?\n\nThe connector needs an instance of a GoogleCredentials in order to connect to the BigQuery APIs. There are multiple\noptions to provide it:\n\n* The default is to load the JSON key from the `GOOGLE_APPLICATION_CREDENTIALS` environment variable, as described\n [here](https://cloud.google.com/docs/authentication/getting-started).\n* In case the environment variable cannot be changed, the credentials file can be configured as\n as a spark option. The file should reside on the same path on all the nodes of the cluster.\n```\n// Globally\nspark.conf.set(\"credentialsFile\", \"\")\n// Per read/Write\nspark.read.format(\"bigquery\").option(\"credentialsFile\", \"\")\n```\n* Credentials can also be provided explicitly, either as a parameter or from Spark runtime configuration.\n They should be passed in as a base64-encoded string directly.\n```\n// Globally\nspark.conf.set(\"credentials\", \"| Property | \nMeaning | \nUsage | \n
|---|---|---|
table\n | \n The BigQuery table in the format [[project:]dataset.]table.\n It is recommended to use the path parameter of\n load()/save() instead. This option has been\n deprecated and will be removed in a future version.\n (Deprecated)\n | \n Read/Write | \n
dataset\n | \n The dataset containing the table. This option should be used with\n standard table and views, but not when loading query results.\n (Optional unless omitted in table)\n | \n Read/Write | \n
project\n | \n The Google Cloud Project ID of the table. This option should be used with\n standard table and views, but not when loading query results.\n (Optional. Defaults to the project of the Service Account being used)\n | \n Read/Write | \n
billingProject\n | \n The Google Cloud Project ID to use for billing (API calls, query execution).\n (Optional. Defaults to the project of the Service Account being used)\n | \n Read/Write | \n
parentProject\n | \n (Deprecated) Alias for billingProject.\n (Optional. Defaults to the project of the Service Account being used)\n | \n Read/Write | \n
location\n | \n The BigQuery location where the data resides (e.g. US, EU, asia-northeast1).\n (Optional. Defaults to BigQuery default)\n | \n Read/Write | \n
maxParallelism\n | \n The maximal number of partitions to split the data into. Actual number\n may be less if BigQuery deems the data small enough. If there are not\n enough executors to schedule a reader per partition, some partitions may\n be empty.\n Important: The old parameter ( parallelism) is\n still supported but in deprecated mode. It will ve removed in\n version 1.0 of the connector.\n (Optional. Defaults to the larger of the preferredMinParallelism and 20,000).)\n | \n Read | \n
preferredMinParallelism\n | \n The preferred minimal number of partitions to split the data into. Actual number\n may be less if BigQuery deems the data small enough. If there are not\n enough executors to schedule a reader per partition, some partitions may\n be empty.\n (Optional. Defaults to the smallest of 3 times the application's default parallelism\n and maxParallelism.)\n | \n Read | \n
viewsEnabled\n | \n Enables the connector to read from views and not only tables. Please read\n the relevant section before activating\n this option.\n (Optional. Defaults to false)\n | \n Read | \n
readDataFormat\n | \n Data Format for reading from BigQuery. Options : ARROW, AVRO\n (Optional. Defaults to ARROW)\n | \n Read | \n
optimizedEmptyProjection\n | \n The connector uses an optimized empty projection (select without any\n columns) logic, used for count() execution. This logic takes\n the data directly from the table metadata or performs a much efficient\n `SELECT COUNT(*) WHERE...` in case there is a filter. You can cancel the\n use of this logic by setting this option to false.\n (Optional, defaults to true)\n | \n Read | \n
pushAllFilters\n | \n If set to true, the connector pushes all the filters Spark can delegate\n to BigQuery Storage API. This reduces amount of data that needs to be sent from\n BigQuery Storage API servers to Spark clients. This option has been\n deprecated and will be removed in a future version.\n (Optional, defaults to true)\n (Deprecated)\n | \n Read | \n
bigQueryJobLabel\n | \n Can be used to add labels to the connector initiated query and load\n BigQuery jobs. Multiple labels can be set.\n (Optional)\n | \n Read | \n
bigQueryTableLabel\n | \n Can be used to add labels to the table while writing to a table. Multiple\n labels can be set.\n (Optional)\n | \n Write | \n
traceApplicationName\n | \n Application name used to trace BigQuery Storage read and write sessions.\n Setting the application name is required to set the trace ID on the\n sessions.\n (Optional)\n | \n Read | \n
traceJobId\n | \n Job ID used to trace BigQuery Storage read and write sessions.\n (Optional, defaults to the Dataproc job ID is exists, otherwise uses\n the Spark application ID)\n | \n Read | \n
createDisposition\n | \n Specifies whether the job is allowed to create new tables. The permitted\n values are:\n
(Optional. Default to CREATE_IF_NEEDED).\n | \n Write | \n
writeMethod\n | \n Controls the method\n in which the data is written to BigQuery. Available values are direct\n to use the BigQuery Storage Write API and indirect which writes the\n data first to GCS and then triggers a BigQuery load operation. See more\n here\n (Optional, defaults to indirect)\n | \n Write | \n
writeAtLeastOnce\n | \n Guarantees that data is written to BigQuery at least once. This is a lesser\n guarantee than exactly once. This is suitable for streaming scenarios\n in which data is continuously being written in small batches.\n (Optional. Defaults to false)\n Supported only by the `DIRECT` write method and mode is NOT `Overwrite`.\n | \n Write | \n
temporaryGcsBucket\n | \n The GCS bucket that temporarily holds the data before it is loaded to\n BigQuery. Required unless set in the Spark configuration\n (spark.conf.set(...)).\n Defaults to the `fs.gs.system.bucket` if exists, for example on Google Cloud Dataproc clusters, starting version 0.42.0.\n Supported only by the `INDIRECT` write method.\n | \n Write | \n
persistentGcsBucket\n | \n The GCS bucket that holds the data before it is loaded to\n BigQuery. If informed, the data won't be deleted after write data\n into BigQuery.\n Supported only by the `INDIRECT` write method.\n | \n Write | \n
persistentGcsPath\n | \n The GCS path that holds the data before it is loaded to\n BigQuery. Used only with persistentGcsBucket.\n Not supported by the `DIRECT` write method.\n | \n Write | \n
intermediateFormat\n | \n The format of the data before it is loaded to BigQuery, values can be\n either \"parquet\",\"orc\" or \"avro\". In order to use the Avro format, the\n spark-avro package must be added in runtime.\n (Optional. Defaults to parquet). On write only. Supported only for the `INDIRECT` write method.\n | \n Write | \n
useAvroLogicalTypes\n | \n When loading from Avro (`.option(\"intermediateFormat\", \"avro\")`), BigQuery uses the underlying Avro types instead of the logical types [by default](https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-avro#logical_types). Supplying this option converts Avro logical types to their corresponding BigQuery data types.\n (Optional. Defaults to false). On write only.\n | \n Write | \n
datePartition\n | \n The date partition the data is going to be written to. Should be a date string\n given in the format YYYYMMDD. Can be used to overwrite the data of\n\t a single partition, like this: (Optional). On write only.\n Can also be used with different partition types like:\n HOUR: YYYYMMDDHH\n MONTH: YYYYMM\n YEAR: YYYY\n Not supported by the `DIRECT` write method.\n | \n Write | \n
partitionField\n | \n If this field is specified, the table is partitioned by this field.\n For Time partitioning, specify together with the option `partitionType`.\n For Integer-range partitioning, specify together with the 3 options: `partitionRangeStart`, `partitionRangeEnd, `partitionRangeInterval`.\n The field must be a top-level TIMESTAMP or DATE field for Time partitioning, or INT64 for Integer-range partitioning. Its mode must be NULLABLE\n or REQUIRED.\n If the option is not set for a Time partitioned table, then the table will be partitioned by pseudo\n column, referenced via either '_PARTITIONTIME' as TIMESTAMP type, or\n '_PARTITIONDATE' as DATE type.\n (Optional).\n Not supported by the `DIRECT` write method.\n | \n Write | \n
partitionExpirationMs\n | \n Number of milliseconds for which to keep the storage for partitions in the table.\n The storage in a partition will have an expiration time of its partition time plus this value.\n (Optional).\n Not supported by the `DIRECT` write method.\n | \n Write | \n
partitionType\n | \n Used to specify Time partitioning.\n Supported types are: HOUR, DAY, MONTH, YEAR\n This option is mandatory for a target table to be Time partitioned.\n (Optional. Defaults to DAY if PartitionField is specified).\n Not supported by the `DIRECT` write method.\n | \n Write | \n
partitionRangeStart,\n partitionRangeEnd,\n partitionRangeInterval\n | \n Used to specify Integer-range partitioning.\n These options are mandatory for a target table to be Integer-range partitioned.\n All 3 options must be specified.\n Not supported by the `DIRECT` write method.\n | \n Write | \n
clusteredFields\n | \n A string of non-repeated, top level columns seperated by comma.\n (Optional).\n | \n Write | \n
allowFieldAddition\n | \n Adds the ALLOW_FIELD_ADDITION\n SchemaUpdateOption to the BigQuery LoadJob. Allowed values are true and false.\n (Optional. Default to false).\n Supported only by the `INDIRECT` write method.\n | \n Write | \n
allowFieldRelaxation\n | \n Adds the ALLOW_FIELD_RELAXATION\n SchemaUpdateOption to the BigQuery LoadJob. Allowed values are true and false.\n (Optional. Default to false).\n Supported only by the `INDIRECT` write method.\n | \n Write | \n
proxyAddress\n | \n Address of the proxy server. The proxy must be a HTTP proxy and address should be in the `host:port` format.\n Can be alternatively set in the Spark configuration (spark.conf.set(...)) or in Hadoop\n Configuration (fs.gs.proxy.address).\n (Optional. Required only if connecting to GCP via proxy.)\n | \n Read/Write | \n
proxyUsername\n | \n The userName used to connect to the proxy. Can be alternatively set in the Spark configuration\n (spark.conf.set(...)) or in Hadoop Configuration (fs.gs.proxy.username).\n (Optional. Required only if connecting to GCP via proxy with authentication.)\n | \n Read/Write | \n
proxyPassword\n | \n The password used to connect to the proxy. Can be alternatively set in the Spark configuration\n (spark.conf.set(...)) or in Hadoop Configuration (fs.gs.proxy.password).\n (Optional. Required only if connecting to GCP via proxy with authentication.)\n | \n Read/Write | \n
httpMaxRetry\n | \n The maximum number of retries for the low-level HTTP requests to BigQuery. Can be alternatively set in the\n Spark configuration (spark.conf.set(\"httpMaxRetry\", ...)) or in Hadoop Configuration\n (fs.gs.http.max.retry).\n (Optional. Default is 10)\n | \n Read/Write | \n
httpConnectTimeout\n | \n The timeout in milliseconds to establish a connection with BigQuery. Can be alternatively set in the\n Spark configuration (spark.conf.set(\"httpConnectTimeout\", ...)) or in Hadoop Configuration\n (fs.gs.http.connect-timeout).\n (Optional. Default is 60000 ms. 0 for an infinite timeout, a negative number for 20000)\n | \n Read/Write | \n
httpReadTimeout\n | \n The timeout in milliseconds to read data from an established connection. Can be alternatively set in the\n Spark configuration (spark.conf.set(\"httpReadTimeout\", ...)) or in Hadoop Configuration\n (fs.gs.http.read-timeout).\n (Optional. Default is 60000 ms. 0 for an infinite timeout, a negative number for 20000)\n | \n Read | \n
arrowCompressionCodec\n | \n Compression codec while reading from a BigQuery table when using Arrow format. Options :\n ZSTD (Zstandard compression),\n LZ4_FRAME (https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md),\n COMPRESSION_UNSPECIFIED. The recommended compression codec is ZSTD\n while using Java.\n (Optional. Defaults to COMPRESSION_UNSPECIFIED which means no compression will be used)\n | \n Read | \n
responseCompressionCodec\n | \n Compression codec used to compress the ReadRowsResponse data. Options:\n RESPONSE_COMPRESSION_CODEC_UNSPECIFIED,\n RESPONSE_COMPRESSION_CODEC_LZ4\n (Optional. Defaults to RESPONSE_COMPRESSION_CODEC_UNSPECIFIED which means no compression will be used)\n | \n Read | \n
cacheExpirationTimeInMinutes\n | \n The expiration time of the in-memory cache storing query information.\n To disable caching, set the value to 0.\n (Optional. Defaults to 15 minutes)\n | \n Read | \n
enableModeCheckForSchemaFields\n | \n Checks the mode of every field in destination schema to be equal to the mode in corresponding source field schema, during DIRECT write.\n Default value is true i.e., the check is done by default. If set to false the mode check is ignored.\n | \n Write | \nenableListInference\n | \n Indicates whether to use schema inference specifically when the mode is Parquet (https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#parquetoptions).\n Defaults to false.\n \n | \n Write | \n \n
bqChannelPoolSize | \n The (fixed) size of the gRPC channel pool created by the BigQueryReadClient.\n For optimal performance, this should be set to at least the number of cores on the cluster executors.\n | \n Read | \n
createReadSessionTimeoutInSeconds\n | \n The timeout in seconds to create a ReadSession when reading a table.\n For Extremely large table this value should be increased.\n (Optional. Defaults to 600 seconds)\n | \n Read | \n
queryJobPriority\n | \n Priority levels set for the job while reading data from BigQuery query. The permitted values are:\n
(Optional. Defaults to INTERACTIVE)\n | \n Read/Write | \n
destinationTableKmsKeyName\n | \n Describes the Cloud KMS encryption key that will be used to protect destination BigQuery\n table. The BigQuery Service Account associated with your project requires access to this\n encryption key. for further Information about using CMEK with BigQuery see\n [here](https://cloud.google.com/bigquery/docs/customer-managed-encryption#key_resource_id).\n Notice: The table will be encrypted by the key only if it created by the\n connector. A pre-existing unencrypted table won't be encrypted just by setting this option.\n (Optional)\n | \n Write | \n
allowMapTypeConversion\n | \n Boolean config to disable conversion from BigQuery records to Spark MapType\n when the record has two subfields with field names as key and value.\n Default value is true which allows the conversion.\n (Optional)\n | \n Read | \n
spark.sql.sources.partitionOverwriteMode\n | \n Config to specify the overwrite mode on write when the table is range/time partitioned.\n Currently supportd two modes : STATIC and DYNAMIC. In STATIC mode,\n the entire table is overwritten. In DYNAMIC mode, the data is overwritten by partitions of the existing table.\n The default value is STATIC.\n (Optional)\n | \n Write | \n
enableReadSessionCaching\n | \n Boolean config to disable read session caching. Caches BigQuery read sessions to allow for faster Spark query planning.\n Default value is true.\n (Optional)\n | \n Read | \n
readSessionCacheDurationMins\n | \n Config to set the read session caching duration in minutes. Only works if enableReadSessionCaching is true (default).\n Allows specifying the duration to cache read sessions for. Maximum allowed value is 300.\n Default value is 5.\n (Optional)\n | \n Read | \n
bigQueryJobTimeoutInMinutes\n | \n Config to set the BigQuery job timeout in minutes.\n Default value is 360 minutes.\n (Optional)\n | \n Read/Write | \n
snapshotTimeMillis\n | \n A timestamp specified in milliseconds to use to read a table snapshot.\n By default this is not set and the latest version of a table is read.\n (Optional)\n | \n Read | \n
bigNumericDefaultPrecision\n | \n An alternative default precision for BigNumeric fields, as the BigQuery default is too wide for Spark. Values can be between 1 and 38.\n This default is used only when the field has an unparameterized BigNumeric type.\n Please note that there might be data loss if the actual data's precision is more than what is specified.\n (Optional)\n | \n Read/Write | \n
bigNumericDefaultScale\n | \n An alternative default scale for BigNumeric fields. Values can be between 0 and 38, and less than bigNumericFieldsPrecision.\n This default is used only when the field has an unparameterized BigNumeric type.\n Please note that there might be data loss if the actual data's scale is more than what is specified.\n (Optional)\n | \n Read/Write | \n
credentialsScopes\n | \n Replaces the scopes of the Google Credentials if the credentials type supports that.\n If scope replacement is not supported then it does nothing.\n The value should be a comma separated list of valid scopes.\n (Optional)\n | \n Read/Write | \n
| BigQuery Standard SQL Data Type \n | \nSpark SQL\n \nData Type\n | \n Notes\n | \n
BOOL\n | \n BooleanType\n | \n \n | \n
INT64\n | \n LongType\n | \n \n | \n
FLOAT64\n | \n DoubleType\n | \n \n | \n
NUMERIC\n | \n DecimalType\n | \n \n Please refer to Numeric and BigNumeric support\n | \n
BIGNUMERIC\n | \n DecimalType\n | \n \n Please refer to Numeric and BigNumeric support\n | \n
STRING\n | \n StringType\n | \n \n | \n
BYTES\n | \n BinaryType\n | \n \n | \n
STRUCT\n | \n StructType\n | \n \n | \n
ARRAY\n | \n ArrayType\n | \n \n | \n
TIMESTAMP\n | \n TimestampType\n | \n \n | \n
DATE\n | \n DateType\n | \n \n | \n
DATETIME\n | \n StringType, TimestampNTZType*\n | \n Spark has no DATETIME type.\n \n Spark string can be written to an existing BQ DATETIME column provided it is in the format for BQ DATETIME literals.\n \n * For Spark 3.4+, BQ DATETIME is read as Spark's TimestampNTZ type i.e. java LocalDateTime\n | \n
TIME\n | \n LongType, StringType*\n | \n Spark has no TIME type. The generated longs, which indicate microseconds since midnight can be safely cast to TimestampType, but this causes the date to be inferred as the current day. Thus times are left as longs and user can cast if they like.\n \nWhen casting to Timestamp TIME have the same TimeZone issues as DATETIME\n \n* Spark string can be written to an existing BQ TIME column provided it is in the format for BQ TIME literals.\n | \n
JSON\n | \n StringType\n | \n Spark has no JSON type. The values are read as String. In order to write JSON back to BigQuery, the following conditions are REQUIRED:\n
| \n
ARRAY<STRUCT<key,value>>\n | \n MapType\n | \n BigQuery has no MAP type, therefore similar to other conversions like Apache Avro and BigQuery Load jobs, the connector converts a Spark Map to a REPEATED STRUCT<key,value>.\n This means that while writing and reading of maps is available, running a SQL on BigQuery that uses map semantics is not supported.\n To refer to the map's values using BigQuery SQL, please check the BigQuery documentation.\n Due to these incompatibilities, a few restrictions apply:\n
| \n
[maxParallelism](#properties) and [preferredMinParallelism](#properties) can be configured explicitly to control the number of partitions.\n\n## Tagging BigQuery Resources\n\nIn order to support tracking the usage of BigQuery resources the connectors\noffers the following options to tag BigQuery resources:\n\n### Adding BigQuery Jobs Labels\n\nThe connector can launch BigQuery load and query jobs. Adding labels to the jobs\nis done in the following manner:\n```\nspark.conf.set(\"bigQueryJobLabel.cost_center\", \"analytics\")\nspark.conf.set(\"bigQueryJobLabel.usage\", \"nightly_etl\")\n```\nThis will create labels `cost_center`=`analytics` and `usage`=`nightly_etl`.\n\n### Adding BigQuery Storage Trace ID\n\nUsed to annotate the read and write sessions. The trace ID is of the format\n`Spark:ApplicationName:JobID`. This is an opt-in option, and to use it the user\nneed to set the `traceApplicationName` property. JobID is auto generated by the\nDataproc job ID, with a fallback to the Spark application ID (such as\n`application_1648082975639_0001`). The Job ID can be overridden by setting the\n`traceJobId` option. Notice that the total length of the trace ID cannot be over\n256 characters.\n\n## Using in Jupyter Notebooks\n\nThe connector can be used in [Jupyter notebooks](https://jupyter.org/) even if\nit is not installed on the Spark cluster. It can be added as an external jar in\nusing the following code:\n\n**Python:**\n```python\nfrom pyspark.sql import SparkSession\nspark = SparkSession.builder \\\n .config(\"spark.jars.packages\", \"com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.44.1\") \\\n .getOrCreate()\ndf = spark.read.format(\"bigquery\") \\\n .load(\"dataset.table\")\n```\n\n**Scala:**\n```scala\nval spark = SparkSession.builder\n.config(\"spark.jars.packages\", \"com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.44.1\")\n.getOrCreate()\nval df = spark.read.format(\"bigquery\")\n.load(\"dataset.table\")\n```\n\nIn case Spark cluster is using Scala 2.12 (it's optional for Spark 2.4.x,\nmandatory in 3.0.x), then the relevant package is\ncom.google.cloud.spark:spark-bigquery-with-dependencies_**2.12**:0.44.1. In\norder to know which Scala version is used, please run the following code:\n\n**Python:**\n```python\nspark.sparkContext._jvm.scala.util.Properties.versionString()\n```\n\n**Scala:**\n```python\nscala.util.Properties.versionString\n```\n## Compiling against the connector\n\nUnless you wish to use the implicit Scala API `spark.read.bigquery(\"TABLE_ID\")`, there is no need to compile against the connector.\n\nTo include the connector in your project:\n\n### Maven\n\n```xml\n| Metric Name | \nDescription | \n
|---|---|
bytes read | \n number of BigQuery bytes read | \n
rows read | \n number of BigQuery rows read | \n
scan time | \n the amount of time spent between read rows response requested to obtained across all the executors, in milliseconds. | \n
parse time | \n the amount of time spent for parsing the rows read across all the executors, in milliseconds. | \n
spark time | \n the amount of time spent in spark to process the queries (i.e., apart from scanning and parsing), across all the executors, in milliseconds. | \n
coalesce on the DataFrame to mitigate this problem.\n\n```\ndesiredPartitionCount = 5\ndfNew = df.coalesce(desiredPartitionCount)\ndfNew.write\n```\n\nA rule of thumb is to have a single partition handle at least 1GB of data.\n\nAlso note that a job running with the `writeAtLeastOnce` property turned on will not encounter CreateWriteStream\nquota errors.\n\n### How do I authenticate outside GCE / Dataproc?\n\nThe connector needs an instance of a GoogleCredentials in order to connect to the BigQuery APIs. There are multiple\noptions to provide it:\n\n* The default is to load the JSON key from the `GOOGLE_APPLICATION_CREDENTIALS` environment variable, as described\n [here](https://cloud.google.com/docs/authentication/getting-started).\n* In case the environment variable cannot be changed, the credentials file can be configured as\n as a spark option. The file should reside on the same path on all the nodes of the cluster.\n```\n// Globally\nspark.conf.set(\"credentialsFile\", \"\")\n// Per read/Write\nspark.read.format(\"bigquery\").option(\"credentialsFile\", \"\")\n```\n* Credentials can also be provided explicitly, either as a parameter or from Spark runtime configuration.\n They should be passed in as a base64-encoded string directly.\n```\n// Globally\nspark.conf.set(\"credentials\", \"public TableInfo createTable(TableId tableId, Schema schema) { return\n * createTable(tableId, schema, Optional.empty()); }\n */\n\n /**\n * Creates an empty table in BigQuery.\n *\n * @param tableId The TableId of the table to be created.\n * @param schema The Schema of the table to be created.\n * @param options Allows configuring the created table\n * @return The {@code Table} object representing the table that was created.\n */\n public TableInfo createTable(TableId tableId, Schema schema, CreateTableOptions options) {\n\n StandardTableDefinition.Builder tableDefinition =\n StandardTableDefinition.newBuilder().setSchema(schema);\n\n options\n .getClusteredFields()\n .ifPresent(\n clusteredFields ->\n tableDefinition.setClustering(\n Clustering.newBuilder().setFields(clusteredFields).build()));\n\n TableInfo.Builder tableInfo = TableInfo.newBuilder(tableId, tableDefinition.build());\n options\n .getKmsKeyName()\n .ifPresent(\n keyName ->\n tableInfo.setEncryptionConfiguration(\n EncryptionConfiguration.newBuilder().setKmsKeyName(keyName).build()));\n if (!options.getBigQueryTableLabels().isEmpty()) {\n tableInfo.setLabels(options.getBigQueryTableLabels());\n }\n return bigQuery.create(tableInfo.build());\n }\n\n /**\n * Creates a temporary table with a job to cleanup after application end, and the same location as\n * the destination table; the temporary table will have the same name as the destination table,\n * with the current time in milliseconds appended to it; useful for holding temporary data in\n * order to overwrite the destination table.\n *\n * @param destinationTableId The TableId of the eventual destination for the data going into the\n * temporary table.\n * @param schema The Schema of the destination / temporary table.\n * @return The {@code Table} object representing the created temporary table.\n */\n public TableInfo createTempTable(TableId destinationTableId, Schema schema) {\n TableId tempTableId = createTempTableId(destinationTableId);\n TableInfo tableInfo =\n TableInfo.newBuilder(tempTableId, StandardTableDefinition.of(schema)).build();\n TableInfo tempTable = bigQuery.create(tableInfo);\n CLEANUP_JOBS.add(() -> deleteTable(tempTable.getTableId()));\n return tempTable;\n }\n\n public TableInfo createTempTableAfterCheckingSchema(\n TableId destinationTableId, Schema schema, boolean enableModeCheckForSchemaFields)\n throws IllegalArgumentException {\n TableInfo destinationTable = getTable(destinationTableId);\n Schema tableSchema = destinationTable.getDefinition().getSchema();\n ComparisonResult schemaWritableResult =\n BigQueryUtil.schemaWritable(\n schema, // sourceSchema\n tableSchema, // destinationSchema\n false, // regardFieldOrder\n enableModeCheckForSchemaFields);\n Preconditions.checkArgument(\n schemaWritableResult.valuesAreEqual(),\n new BigQueryConnectorException.InvalidSchemaException(\n \"Destination table's schema is not compatible with dataframe's schema. \"\n + schemaWritableResult.makeMessage()));\n return createTempTable(destinationTableId, schema);\n }\n\n public TableId createTempTableId(TableId destinationTableId) {\n String tempProject = materializationProject.orElseGet(destinationTableId::getProject);\n String tempDataset = materializationDataset.orElseGet(destinationTableId::getDataset);\n String tableName = destinationTableId.getTable() + System.nanoTime();\n TableId tempTableId =\n tempProject == null\n ? TableId.of(tempDataset, tableName)\n : TableId.of(tempProject, tempDataset, tableName);\n return tempTableId;\n }\n\n /**\n * Deletes this table in BigQuery.\n *\n * @param tableId The TableId of the table to be deleted.\n * @return True if the operation was successful, false otherwise.\n */\n public boolean deleteTable(TableId tableId) {\n log.info(\"Deleting table \" + fullTableName(tableId));\n return bigQuery.delete(tableId);\n }\n\n private Job copyData(\n TableId sourceTableId,\n TableId destinationTableId,\n JobInfo.WriteDisposition writeDisposition) {\n String queryFormat = \"SELECT * FROM `%s`\";\n String temporaryTableName = fullTableName(sourceTableId);\n String sqlQuery = String.format(queryFormat, temporaryTableName);\n QueryJobConfiguration queryConfig =\n jobConfigurationFactory\n .createQueryJobConfigurationBuilder(sqlQuery, Collections.emptyMap())\n .setUseLegacySql(false)\n .setDestinationTable(destinationTableId)\n .setWriteDisposition(writeDisposition)\n .build();\n\n return create(JobInfo.newBuilder(queryConfig).build());\n }\n\n public boolean isTablePartitioned(TableId tableId) {\n TableInfo table = getTable(tableId);\n if (table == null) {\n return false;\n }\n TableDefinition tableDefinition = table.getDefinition();\n if (tableDefinition instanceof StandardTableDefinition) {\n StandardTableDefinition sdt = (StandardTableDefinition) tableDefinition;\n return sdt.getTimePartitioning() != null || sdt.getRangePartitioning() != null;\n }\n return false;\n }\n\n /**\n * Overwrites the partitions of the destination table, using the partitions from the given\n * temporary table, transactionally.\n *\n * @param temporaryTableId The {@code TableId} representing the temporary-table.\n * @param destinationTableId The {@code TableId} representing the destination table.\n * @return The {@code Job} object representing this operation (which can be tracked to wait until\n * it has finished successfully).\n */\n public Job overwriteDestinationWithTemporaryDynamicPartitons(\n TableId temporaryTableId, TableId destinationTableId) {\n\n TableDefinition destinationDefinition = getTable(destinationTableId).getDefinition();\n String sqlQuery = null;\n if (destinationDefinition instanceof StandardTableDefinition) {\n String destinationTableName = fullTableName(destinationTableId);\n String temporaryTableName = fullTableName(temporaryTableId);\n StandardTableDefinition sdt = (StandardTableDefinition) destinationDefinition;\n\n TimePartitioning timePartitioning = sdt.getTimePartitioning();\n if (timePartitioning != null) {\n sqlQuery =\n getQueryForTimePartitionedTable(\n destinationTableName, temporaryTableName, sdt, timePartitioning);\n } else {\n RangePartitioning rangePartitioning = sdt.getRangePartitioning();\n if (rangePartitioning != null) {\n sqlQuery =\n getQueryForRangePartitionedTable(\n destinationTableName, temporaryTableName, sdt, rangePartitioning);\n }\n }\n\n if (sqlQuery != null) {\n QueryJobConfiguration queryConfig =\n jobConfigurationFactory\n .createQueryJobConfigurationBuilder(sqlQuery, Collections.emptyMap())\n .setUseLegacySql(false)\n .build();\n\n return create(JobInfo.newBuilder(queryConfig).build());\n }\n }\n\n // no partitioning default to statndard overwrite\n return overwriteDestinationWithTemporary(temporaryTableId, destinationTableId);\n }\n\n /**\n * Overwrites the given destination table, with all the data from the given temporary table,\n * transactionally.\n *\n * @param temporaryTableId The {@code TableId} representing the temporary-table.\n * @param destinationTableId The {@code TableId} representing the destination table.\n * @return The {@code Job} object representing this operation (which can be tracked to wait until\n * it has finished successfully).\n */\n public Job overwriteDestinationWithTemporary(\n TableId temporaryTableId, TableId destinationTableId) {\n String queryFormat =\n \"MERGE `%s`\\n\"\n + \"USING (SELECT * FROM `%s`)\\n\"\n + \"ON FALSE\\n\"\n + \"WHEN NOT MATCHED THEN INSERT ROW\\n\"\n + \"WHEN NOT MATCHED BY SOURCE THEN DELETE\";\n\n String destinationTableName = fullTableName(destinationTableId);\n String temporaryTableName = fullTableName(temporaryTableId);\n String sqlQuery = String.format(queryFormat, destinationTableName, temporaryTableName);\n QueryJobConfiguration queryConfig =\n jobConfigurationFactory\n .createQueryJobConfigurationBuilder(sqlQuery, Collections.emptyMap())\n .setUseLegacySql(false)\n .build();\n\n return create(JobInfo.newBuilder(queryConfig).build());\n }\n\n /**\n * Appends all the data from the given temporary table, to the given destination table,\n * transactionally.\n *\n * @param temporaryTableId The {@code TableId} representing the temporary-table.\n * @param destinationTableId The {@code TableId} representing the destination table.\n * @return The {@code Job} object representing this operation (which can be tracked to wait until\n * it has finished successfully).\n */\n public Job appendDestinationWithTemporary(TableId temporaryTableId, TableId destinationTableId) {\n return copyData(temporaryTableId, destinationTableId, JobInfo.WriteDisposition.WRITE_APPEND);\n }\n\n /**\n * Creates a String appropriately formatted for BigQuery Storage Write API representing the given\n * table.\n *\n * @param tableId The {@code TableId} representing the given object.\n * @return The formatted String.\n */\n public String createTablePathForBigQueryStorage(TableId tableId) {\n Preconditions.checkNotNull(tableId, \"tableId cannot be null\");\n // We need the full path for the createWriteStream method. We used to have it by creating the\n // table and then taking its full tableId, but that caused an issue with the ErrorIfExists\n // implementation (now the check, done in another place is positive). To solve it, we do what\n // the BigQuery client does on Table ID with no project - take the BigQuery client own project\n // ID. This gives us the same behavior but allows us to defer the table creation to the last\n // minute.\n String project = tableId.getProject() != null ? tableId.getProject() : getProjectId();\n return String.format(\n \"projects/%s/datasets/%s/tables/%s\", project, tableId.getDataset(), tableId.getTable());\n }\n\n public TableInfo getReadTable(ReadTableOptions options) {\n Optional listTables(DatasetId datasetId, TableDefinition.Type... types) {\n Set
allTables = bigQuery.listTables(datasetId).iterateAll();\n return StreamSupport.stream(allTables.spliterator(), false)\n .filter(table -> allowedTypes.contains(table.getDefinition().getType()))\n .collect(ImmutableList.toImmutableList());\n }\n\n public Table update(TableInfo table) {\n return bigQuery.update(table);\n }\n\n public Job createAndWaitFor(JobConfiguration.Builder jobConfiguration) {\n return createAndWaitFor(jobConfiguration.build());\n }\n\n public Job createAndWaitFor(JobConfiguration jobConfiguration) {\n JobInfo jobInfo = JobInfo.of(jobConfiguration);\n Job job = bigQuery.create(jobInfo);\n Job returnedJob = null;\n\n log.info(\"Submitted job {}. jobId: {}\", jobConfiguration, job.getJobId());\n try {\n Job completedJob = job.waitFor();\n if (completedJob == null) {\n throw new BigQueryException(\n BaseHttpServiceException.UNKNOWN_CODE,\n String.format(\"Failed to run the job [%s], got null back\", job));\n }\n if (completedJob.getStatus().getError() != null) {\n throw new BigQueryException(\n BaseHttpServiceException.UNKNOWN_CODE,\n String.format(\n \"Failed to run the job [%s], due to '%s'\", completedJob.getStatus().getError()));\n }\n return completedJob;\n } catch (InterruptedException e) {\n Thread.currentThread().interrupt();\n throw new BigQueryException(\n BaseHttpServiceException.UNKNOWN_CODE,\n String.format(\"Failed to run the job [%s], task was interrupted\", job),\n e);\n }\n }\n\n Job create(JobInfo jobInfo) {\n return bigQuery.create(jobInfo);\n }\n\n public TableResult query(String sql) {\n try {\n return bigQuery.query(\n jobConfigurationFactory\n .createQueryJobConfigurationBuilder(sql, Collections.emptyMap())\n .build());\n } catch (InterruptedException e) {\n Thread.currentThread().interrupt();\n throw new BigQueryException(\n BaseHttpServiceException.UNKNOWN_CODE,\n String.format(\"Failed to run the query [%s]\", sql),\n e);\n }\n }\n\n String createSql(\n TableId table,\n ImmutableList