Updated All Externally Visible Logs/Docs from Runner V2 to Portable Runner #38532
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -244,6 +244,40 @@ func TestGetJobOptions_DisableRunnerV2ExperimentsSet(t *testing.T) { | |
| } | ||
| } | ||
|
|
||
| func TestGetJobOptions_DisablePortableRunnerExperimentsSet(t *testing.T) { | ||
| resetGlobals() | ||
| *stagingLocation = "gs://testStagingLocation" | ||
| *gcpopts.Project = "testProject" | ||
| *gcpopts.Region = "testRegion" | ||
| *jobopts.Experiments = "disable_portable_runner" | ||
|
|
||
| opts, err := getJobOptions(context.Background(), false) | ||
|
|
||
| if err == nil { | ||
|
Contributor
There was a problem hiding this comment. what are these testing? It seems it is verifying errors, but the test name doesn't indicate we expect a failure. If we do it woudl be better to verify the error more specifically too ditto for the other test |
||
| t.Error("getJobOptions() returned error nil, want an error") | ||
| } | ||
| if opts != nil { | ||
| t.Errorf("getJobOptions() returned JobOptions when it should not have, got %#v, want nil", opts) | ||
| } | ||
| } | ||
|
|
||
| func TestGetJobOptions_EnableStreamingJavaRunnerExperimentsSet(t *testing.T) { | ||
| resetGlobals() | ||
| *stagingLocation = "gs://testStagingLocation" | ||
| *gcpopts.Project = "testProject" | ||
| *gcpopts.Region = "testRegion" | ||
| *jobopts.Experiments = "enable_streaming_java_runner" | ||
|
|
||
| opts, err := getJobOptions(context.Background(), false) | ||
|
|
||
| if err == nil { | ||
| t.Error("getJobOptions() returned error nil, want an error") | ||
| } | ||
| if opts != nil { | ||
| t.Errorf("getJobOptions() returned JobOptions when it should not have, got %#v, want nil", opts) | ||
| } | ||
| } | ||
|
|
||
| func TestGetJobOptions_NoStagingLocation(t *testing.T) { | ||
| resetGlobals() | ||
| *stagingLocation = "" | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -135,7 +135,7 @@ def _state_read_range(self, buffer_state, range_lo, range_hi): | |
|
|
||
| def _state_clear_range(self, buffer_state, range_lo, range_hi): | ||
| """Clears a specified range of elements from the buffer state.""" | ||
| # TODO: Dataflow Portable Runner gets stuck when MIN_TIMESTAMP is used | ||
| # as the lower bound for clear_range. Investigate this further. | ||
| buffer_state.clear_range(range_lo, range_hi) | ||
|
|
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -67,7 +67,7 @@ Perform Beam runner specific setup. | |
|
|
||
| ℹ️ Note that cross-language transforms require | ||
| portable implementations of Spark/Flink/Direct runners. Dataflow requires | ||
| [Portable Runner](https://cloud.google.com/dataflow/docs/guides/deploying-a-pipeline#dataflow-runner-v2). | ||
|
Contributor
There was a problem hiding this comment. how about |
||
| See [here](https://beam.apache.org/documentation/runners/dataflow/) for | ||
| instructions for setting up Dataflow. | ||
|
|
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -563,7 +563,7 @@ def test_streaming_with_fixed_num_streams(self): | |
|
|
||
| @unittest.skip( | ||
| "Streaming to the Storage Write API sink with autosharding is broken " | ||
| "with Dataflow Portable Runner.") | ||
| def test_streaming_with_auto_sharding(self): | ||
| self.skip_if_not_dataflow_runner() | ||
| table = 'streaming_with_auto_sharding' | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -103,7 +103,7 @@ def _translate_step_name(self, internal_name): | |
| user_step_name = None | ||
| if (self._job_graph and internal_name | ||
| in self._job_graph.proto_pipeline.components.transforms.keys()): | ||
| # Dataflow Portable Runner with portable job submission uses proto transform map | ||
| # IDs for step names. Also PTransform.unique_name maps to user step names. | ||
| # Hence we lookup user step names based on the proto. | ||
| user_step_name = self._job_graph.proto_pipeline.components.transforms[ | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -65,7 +65,7 @@ | |
|
|
||
| BQ_SOURCE_UW_ERROR = ( | ||
| 'The Read(BigQuerySource(...)) transform is not supported with newer stack ' | ||
| 'features (Fn API, Dataflow Portable Runner, etc). Please use the transform ' | ||
| 'apache_beam.io.gcp.bigquery.ReadFromBigQuery instead.') | ||
|
|
||
|
|
||
|
|
@@ -320,7 +320,7 @@ def visit_transform(self, applied_transform): | |
| raise ValueError( | ||
| 'CombineFn.setup and CombineFn.teardown are ' | ||
| 'not supported with non-portable Dataflow ' | ||
| 'runner. Please use Dataflow Portable Runner instead.') | ||
|
|
||
| @staticmethod | ||
| def _overrides_setup_or_teardown(combinefn): | ||
|
|
@@ -342,7 +342,7 @@ def run_pipeline(self, pipeline, options, pipeline_proto=None): | |
| """Remotely executes entire pipeline or parts reachable from node.""" | ||
| if _is_runner_v2_disabled(options): | ||
| raise ValueError( | ||
| 'Disabling Portable Runner no longer supported ' | ||
| 'using Beam Python %s.' % beam.version.__version__) | ||
|
|
||
| # Label goog-dataflow-notebook if job is started from notebook. | ||
|
|
@@ -591,6 +591,8 @@ def _add_runner_v2_missing_options(options): | |
| debug_options.add_experiment('use_unified_worker') | ||
| debug_options.add_experiment('use_runner_v2') | ||
| debug_options.add_experiment('use_portable_job_submission') | ||
| # enable_portable_runner is not added by default as it is not documented. | ||
| # This behavior will be fixed in later versions. | ||
|
|
||
|
|
||
| def _check_and_add_missing_options(options): | ||
|
|
@@ -648,8 +650,8 @@ def _check_and_add_missing_streaming_options(options): | |
|
|
||
| :param options: PipelineOptions for this pipeline. | ||
| """ | ||
| # Streaming only supports using Portable Runner (aka unified worker). | ||
| # Portable Runner only supports using streaming engine (aka windmill service) | ||
| if options.view_as(StandardOptions).streaming: | ||
| debug_options = options.view_as(DebugOptions) | ||
| debug_options.add_experiment('enable_streaming_engine') | ||
|
|
@@ -659,9 +661,11 @@ def _check_and_add_missing_streaming_options(options): | |
| def _is_runner_v2_disabled(options): | ||
| # Type: (PipelineOptions) -> bool | ||
|
|
||
| """Returns true if Portable Runner is disabled.""" | ||
| debug_options = options.view_as(DebugOptions) | ||
| return ( | ||
| debug_options.lookup_experiment('disable_portable_runner') or | ||
| debug_options.lookup_experiment('enable_streaming_java_runner') or | ||
| debug_options.lookup_experiment('disable_runner_v2') or | ||
| debug_options.lookup_experiment('disable_runner_v2_until_2023') or | ||
| debug_options.lookup_experiment('disable_runner_v2_until_v2.50') or | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -41,6 +41,7 @@ | |
| from apache_beam.runners.dataflow.dataflow_runner import DataflowRuntimeException | ||
| from apache_beam.runners.dataflow.dataflow_runner import _check_and_add_missing_options | ||
| from apache_beam.runners.dataflow.dataflow_runner import _check_and_add_missing_streaming_options | ||
| from apache_beam.runners.dataflow.dataflow_runner import _is_runner_v2_disabled | ||
| from apache_beam.runners.dataflow.internal.clients import dataflow as dataflow_api | ||
| from apache_beam.runners.internal import names | ||
| from apache_beam.runners.runner import PipelineState | ||
|
|
@@ -733,6 +734,24 @@ def test_explicit_streaming_no_unbounded(self): | |
| p.result.job.proto.type, | ||
| apiclient.dataflow.Job.TypeValueValuesEnum.JOB_TYPE_STREAMING) | ||
|
|
||
| def test_runner_v2_disabled_experiments_raise(self): | ||
| disable_experiments = [ | ||
| 'disable_portable_runner', | ||
| 'enable_streaming_java_runner', | ||
| 'disable_runner_v2', | ||
| 'disable_runner_v2_until_2023', | ||
| 'disable_runner_v2_until_v2.50', | ||
| 'disable_prime_runner_v2', | ||
| ] | ||
| for experiment in disable_experiments: | ||
| options = PipelineOptions([f'--experiments={experiment}']) | ||
| self.assertTrue( | ||
| _is_runner_v2_disabled(options), | ||
| f'Expected {experiment} to disable Portable Runner') | ||
| with self.assertRaisesRegex(ValueError, | ||
| 'Disabling Portable Runner .* no longer supported'): | ||
| DataflowRunner().run_pipeline(None, options) | ||
|
|
||
|
|
||
| if __name__ == '__main__': | ||
| unittest.main() | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1153,7 +1153,7 @@ def to_split_int(n): | |
|
|
||
|
|
||
| # TODO: Used in legacy batch worker. Move under MetricUpdateTranslators | ||
| # after Portable Runner transition. | ||
| def translate_distribution(distribution_update, metric_update_proto): | ||
| """Translate metrics DistributionUpdate to dataflow distribution update. | ||
|
|
||
|
|
@@ -1174,7 +1174,7 @@ def translate_distribution(distribution_update, metric_update_proto): | |
| metric_update_proto.distribution = dist_update_proto | ||
|
|
||
|
|
||
| # TODO: Used in legacy batch worker. Delete after Portable Runner transition. | ||
| def translate_value(value, metric_update_proto): | ||
| metric_update_proto.integer = to_split_int(value) | ||
|
|
||
|
|
@@ -1203,8 +1203,8 @@ def get_container_image_from_options(pipeline_options): | |
| if worker_options.sdk_container_image: | ||
| return worker_options.sdk_container_image | ||
|
|
||
| # Legacy and Portable Runner exist in different repositories. | ||
| # Set to legacy format, override if Portable Runner | ||
| container_repo = names.DATAFLOW_CONTAINER_IMAGE_REPOSITORY | ||
| image_name = '{repository}/beam_python{major}.{minor}_sdk'.format( | ||
| repository=container_repo, | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -6764,9 +6764,9 @@ class StreamingConfigTask(_messages.Message): | |
| format version for streaming engine jobs. | ||
| userStepToStateFamilyNameMap: Map from user step names to state families. | ||
| userWorkerRunnerV1Settings: Binary encoded proto to control runtime | ||
| behavior of the Streaming Java Runner user worker. | ||
|
Contributor
There was a problem hiding this comment. think this file is generated |
||
| userWorkerRunnerV2Settings: Binary encoded proto to control runtime | ||
| behavior of the Portable Runner user worker. | ||
| windmillServiceEndpoint: If present, the worker must use this endpoint to | ||
| communicate with Windmill Service dispatchers, otherwise the worker must | ||
| continue to use whatever endpoint it had been using. | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1096,7 +1096,7 @@ def setup(self, *args, **kwargs): | |
| before executing any of the other methods. The resources can then be | ||
| disposed of in ``CombineFn.teardown``. | ||
|
|
||
| If you are using Dataflow, you need to enable Dataflow Portable Runner | ||
| before using this feature. | ||
|
|
||
| Args: | ||
|
|
@@ -1194,7 +1194,7 @@ def extract_output(self, accumulator, *args, **kwargs): | |
| def teardown(self, *args, **kwargs): | ||
| """Called to clean up an instance before it is discarded. | ||
|
|
||
| If you are using Dataflow, you need to enable Dataflow Portable Runner | ||
| before using this feature. | ||
|
|
||
| Args: | ||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The word "streaming" is redundant here as "Streaming Java Runner" already includes it.