Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CDPD-12081 Upgrade gcs connector to v1.9.17 from upstream in HDP 3.1-maint #8

Open
wants to merge 69 commits into
base: HDP-3.1-maint
Choose a base branch
from

Commits on Nov 1, 2018

  1. Configuration menu
    Copy the full SHA
    1447c6c View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2018

  1. Use assertThrows where possible #cleanup

    	Change on 2018/11/05 by idv <idv@google.com>
    
    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=220160326
    medb committed Nov 6, 2018
    Configuration menu
    Copy the full SHA
    7babc9b View commit details
    Browse the repository at this point in the history
  2. Use 'uri-path' as the default path codec.

    	Change on 2018/11/05 by idv <idv@google.com>
    
    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=220208182
    medb committed Nov 6, 2018
    Configuration menu
    Copy the full SHA
    3608d4c View commit details
    Browse the repository at this point in the history

Commits on Dec 6, 2018

  1. Fix GCSIO integration test.

    Apparently, somebody created "does-not-exist" GCS bucket that caused one of the tests (that expects that this bucket does not exist!) to fail.
    
    	Change on 2018/12/05 by idv <idv@google.com>
    
    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=224261022
    medb committed Dec 6, 2018
    Configuration menu
    Copy the full SHA
    2f580f8 View commit details
    Browse the repository at this point in the history
  2. Update GCS connector dependencies to latest versions.

    	Change on 2018/12/05 by idv <idv@google.com>
    
    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=224262503
    medb committed Dec 6, 2018
    Configuration menu
    Copy the full SHA
    078cc17 View commit details
    Browse the repository at this point in the history

Commits on Dec 11, 2018

  1. Parallelize Maven builds

    	Change on 2018/12/10 by idv <idv@google.com>
    
    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=224891096
    medb committed Dec 11, 2018
    Configuration menu
    Copy the full SHA
    cb07768 View commit details
    Browse the repository at this point in the history
  2. Prefetch metadata with only 1 list request.

    	Change on 2018/12/10 by idv <idv@google.com>
    
    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=224916690
    medb committed Dec 11, 2018
    Configuration menu
    Copy the full SHA
    e522221 View commit details
    Browse the repository at this point in the history

Commits on Dec 14, 2018

  1. Support lazy initialization of GHFS

    	Change on 2018/12/13 by idv <idv@google.com>
    
    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=225450649
    medb committed Dec 14, 2018
    Configuration menu
    Copy the full SHA
    8978a99 View commit details
    Browse the repository at this point in the history

Commits on Dec 15, 2018

  1. Configuration menu
    Copy the full SHA
    eb8e5a9 View commit details
    Browse the repository at this point in the history

Commits on Dec 17, 2018

  1. Remove redundant log message before thrown exception in ResilientOper…

    …ation
    
    	Change on 2018/12/14 by idv <idv@google.com>
    
    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=225564622
    medb committed Dec 17, 2018
    Configuration menu
    Copy the full SHA
    5068483 View commit details
    Browse the repository at this point in the history
  2. Clean up.

    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=225887390
    medb authored and functicons committed Dec 17, 2018
    Configuration menu
    Copy the full SHA
    0bfa68a View commit details
    Browse the repository at this point in the history

Commits on Dec 20, 2018

  1. Configuration menu
    Copy the full SHA
    15616bf View commit details
    Browse the repository at this point in the history
  2. Minor clean up

    	Change on 2018/12/18 by idv <idv@google.com>
    
    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=226058870
    medb authored and functicons committed Dec 20, 2018
    Configuration menu
    Copy the full SHA
    c9088d9 View commit details
    Browse the repository at this point in the history
  3. LSC: Remove or broaden the visibility of @VisibleForTesting annotatio…

    …ns which are accessed from production code.
    
    These annotations are being violated; they are being removed to allow the introduction of an ErrorProne check which highlights such errors.
    
    More information: []
    
    Tested:
        TAP --sample for global presubmit queue
        []
    	Change on 2018/12/19 by ghm <ghm@google.com>
    
    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=226249949
    graememorgan authored and functicons committed Dec 20, 2018
    Configuration menu
    Copy the full SHA
    2392540 View commit details
    Browse the repository at this point in the history
  4. Fix compilation failures

    medb committed Dec 20, 2018
    Configuration menu
    Copy the full SHA
    fc5b370 View commit details
    Browse the repository at this point in the history
  5. Release GCS connector 1.9.11 and BQ connector 0.13.11.

    	Change on 2018/12/20 by dagang <dagang@google.com>
    
    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=226362653
    dagang authored and medb committed Dec 20, 2018
    Configuration menu
    Copy the full SHA
    9d77d19 View commit details
    Browse the repository at this point in the history
  6. Update versions for next connectors release development

    	Change on 2018/12/20 by dagang <dagang@google.com>
    
    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=226393561
    dagang authored and functicons committed Dec 20, 2018
    Configuration menu
    Copy the full SHA
    03a23eb View commit details
    Browse the repository at this point in the history
  7. Add Maven Central badges

    medb authored Dec 20, 2018
    Configuration menu
    Copy the full SHA
    f7c90df View commit details
    Browse the repository at this point in the history

Commits on Dec 21, 2018

  1. Configuration menu
    Copy the full SHA
    8256670 View commit details
    Browse the repository at this point in the history
  2. Add LGTM badges

    medb authored Dec 21, 2018
    Configuration menu
    Copy the full SHA
    c2df8f3 View commit details
    Browse the repository at this point in the history

Commits on Dec 23, 2018

  1. Add more badges

    medb authored Dec 23, 2018
    Configuration menu
    Copy the full SHA
    7b52cf8 View commit details
    Browse the repository at this point in the history

Commits on Dec 24, 2018

  1. Remove redundant license badge

    medb authored Dec 24, 2018
    Configuration menu
    Copy the full SHA
    0cb3521 View commit details
    Browse the repository at this point in the history

Commits on Dec 25, 2018

  1. Remove redundant lgtm.com badge

    medb authored Dec 25, 2018
    Configuration menu
    Copy the full SHA
    9e572ec View commit details
    Browse the repository at this point in the history

Commits on Dec 28, 2018

  1. Fix GCS connector documentation.

    	Change on 2018/12/28 by idv <idv@google.com>
    
    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=227151680
    medb committed Dec 28, 2018
    Configuration menu
    Copy the full SHA
    dafb138 View commit details
    Browse the repository at this point in the history

Commits on Jan 3, 2019

  1. Improve exception message

    Fixes: GoogleCloudDataproc#22
    
    	Change on 2019/01/03 by idv <idv@google.com>
    
    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=227718811
    medb committed Jan 3, 2019
    Configuration menu
    Copy the full SHA
    ab9b5ea View commit details
    Browse the repository at this point in the history
  2. Fix partial read if exception is thrown on last retry.

    	Change on 2019/01/03 by idv <idv@google.com>
    
    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=227762104
    medb committed Jan 3, 2019
    Configuration menu
    Copy the full SHA
    55528bc View commit details
    Browse the repository at this point in the history

Commits on Jan 16, 2019

  1. Add info logging for failed delete and rename operations.

    Customers report their jobs frequently fail because of failed rename and delete operations, but it's impossible to diagnose them, because nothing is logged by default.
    
    	Change on 2019/01/16 by idv <idv@google.com>
    
    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=229592005
    medb committed Jan 16, 2019
    Configuration menu
    Copy the full SHA
    6ef6121 View commit details
    Browse the repository at this point in the history

Commits on Jan 18, 2019

  1. Configuration menu
    Copy the full SHA
    0e30d8a View commit details
    Browse the repository at this point in the history

Commits on Jan 26, 2019

  1. Update connectors dependencies.

    	Change on 2019/01/25 by idv <idv@google.com>
    
    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=230956801
    medb committed Jan 26, 2019
    Configuration menu
    Copy the full SHA
    6600463 View commit details
    Browse the repository at this point in the history
  2. Improve exception message.

    	Change on 2019/01/25 by idv <idv@google.com>
    
    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=230994471
    medb committed Jan 26, 2019
    Configuration menu
    Copy the full SHA
    5a9ed1c View commit details
    Browse the repository at this point in the history
  3. Minor fixes to read logic.

    1. Correctly initialize contentChannelEnd when reading gzipped file from the beginning.
    2. Exit from a read loop if reached object end.
    
    	Change on 2019/01/25 by idv <idv@google.com>
    
    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=230998166
    medb committed Jan 26, 2019
    Configuration menu
    Copy the full SHA
    7a7f06e View commit details
    Browse the repository at this point in the history
  4. #cleanup test failure message formatting

    	Change on 2019/01/25 by idv <idv@google.com>
    
    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=231004001
    medb committed Jan 26, 2019
    Configuration menu
    Copy the full SHA
    7713969 View commit details
    Browse the repository at this point in the history

Commits on Jan 28, 2019

  1. Fix for error-prone UnnecessaryParentheses warning

    	Change on 2019/01/28 by idv <idv@google.com>
    
    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=231227113
    medb committed Jan 28, 2019
    Configuration menu
    Copy the full SHA
    75ce5ee View commit details
    Browse the repository at this point in the history
  2. Roll-back Apache HTTP Client version to fix integration tests.

    	Change on 2019/01/28 by idv <idv@google.com>
    
    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=231241840
    medb committed Jan 28, 2019
    Configuration menu
    Copy the full SHA
    0a6a927 View commit details
    Browse the repository at this point in the history
  3. Improve GCS IO exception messages.

    	Change on 2019/01/28 by idv <idv@google.com>
    
    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=231281876
    medb committed Jan 28, 2019
    Configuration menu
    Copy the full SHA
    fcc8ede View commit details
    Browse the repository at this point in the history

Commits on Jan 29, 2019

  1. Initial set for location restriction. Since the regions do not exist …

    …yet, we are only testing for correctness of annotation.
    
    	Change on 2019/01/28 by sedov <sedov@google.com>
    
    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=231313844
    DanSedov authored and medb committed Jan 29, 2019
    Configuration menu
    Copy the full SHA
    1942282 View commit details
    Browse the repository at this point in the history
  2. Improve testMultipleDeleteBucket test case.

    	Change on 2019/01/29 by idv <idv@google.com>
    
    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=231419710
    medb committed Jan 29, 2019
    Configuration menu
    Copy the full SHA
    6b28380 View commit details
    Browse the repository at this point in the history
  3. Parallelize get and list GCS requests and remove redundant GCS reques…

    …ts in GCS FS methods.
    
    	Change on 2019/01/29 by idv <idv@google.com>
    
    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=231424895
    medb committed Jan 29, 2019
    Configuration menu
    Copy the full SHA
    a1673fd View commit details
    Browse the repository at this point in the history

Commits on Jan 30, 2019

  1. Fix bug that could lead to data duplication when reading files with G…

    …ZIP content encoding.
    
    	Change on 2019/01/30 by idv <idv@google.com>
    
    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=231649255
    medb committed Jan 30, 2019
    Configuration menu
    Copy the full SHA
    a711b8f View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    8d17fe9 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    adf9311 View commit details
    Browse the repository at this point in the history

Commits on Feb 4, 2019

  1. Fix directory inference.

    Inferred directory should always have directory path (ends with "/").
    
    	Change on 2019/02/04 by idv <idv@google.com>
    
    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=232300172
    medb committed Feb 4, 2019
    Configuration menu
    Copy the full SHA
    52f5055 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    142c51b View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    e564678 View commit details
    Browse the repository at this point in the history

Commits on Feb 8, 2019

  1. Configuration menu
    Copy the full SHA
    dcae140 View commit details
    Browse the repository at this point in the history
  2. Clean up Markdown formatting [skip ci]

    * use Markdown formatting instead of HTML formatting
    * added link to search for all questions tagged with `google-cloud-dataproc` on SO
    mbrukman authored and medb committed Feb 8, 2019
    Configuration menu
    Copy the full SHA
    d3f61ab View commit details
    Browse the repository at this point in the history
  3. Improve README files

    medb committed Feb 8, 2019
    Configuration menu
    Copy the full SHA
    f33f594 View commit details
    Browse the repository at this point in the history
  4. Fix usages of any(<Primitive>.class) matchers

    In Mockito 2, if a method expects a primitive type, but an any(<Primitive>.class) matcher is used in its place, it will throw an error. To prepare for this upcoming breakage, change
    all existing any(<Primitive>.class) matchers to use the correct any<Primitive>() matcher.
    
    For more information see []
    Tested:
        TAP --sample for global presubmit queue
        []
    	Change on 2019/02/04 by tvanderlippe <tvanderlippe@google.com>
    
    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=232301764
    TimvdLippe authored and medb committed Feb 8, 2019
    Configuration menu
    Copy the full SHA
    5cabe18 View commit details
    Browse the repository at this point in the history

Commits on Feb 11, 2019

  1. Configuration menu
    Copy the full SHA
    d9ea4b8 View commit details
    Browse the repository at this point in the history
  2. Use daemon threads for concurrent globbing.

    This will prevent Spark jobs from hanging after all work is done.
    
    Fixes GoogleCloudDataproc#150:
    GoogleCloudDataproc#150
    
    	Change on 2019/02/11 by idv <idv@google.com>
    
    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=233437145
    medb committed Feb 11, 2019
    Configuration menu
    Copy the full SHA
    3729769 View commit details
    Browse the repository at this point in the history

Commits on Feb 14, 2019

  1. Add Hadoop File System extended attributes support.

    	Change on 2019/02/13 by idv <idv@google.com>
    
    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=233877650
    medb committed Feb 14, 2019
    Configuration menu
    Copy the full SHA
    02b17d1 View commit details
    Browse the repository at this point in the history
  2. Fix Hadoop 1 tests

    medb committed Feb 14, 2019
    Configuration menu
    Copy the full SHA
    4a428d5 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    e7dfbd9 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    4e097f2 View commit details
    Browse the repository at this point in the history

Commits on Feb 21, 2019

  1. Configuration menu
    Copy the full SHA
    c920e4e View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    1550a05 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    c2c14e3 View commit details
    Browse the repository at this point in the history

Commits on Feb 23, 2019

  1. Configuration menu
    Copy the full SHA
    137337c View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    9e4ace2 View commit details
    Browse the repository at this point in the history

Commits on Feb 24, 2019

  1. Configuration menu
    Copy the full SHA
    f81d823 View commit details
    Browse the repository at this point in the history

Commits on Feb 25, 2019

  1. Eagerly fetch GoogleCloudStorageReadChannel metadata if 'fs.gs.inputs…

    …tream.fast.fail.on.not.found.enable' is true
    medb committed Feb 25, 2019
    Configuration menu
    Copy the full SHA
    bde7be4 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    6abbb16 View commit details
    Browse the repository at this point in the history

Commits on May 15, 2019

  1. Configuration menu
    Copy the full SHA
    dd6b3cc View commit details
    Browse the repository at this point in the history
  2. BigQuery connector: support nested record type in field schema.

    Fixes GoogleCloudDataproc#144
    
    	Change on 2019/04/26 by dagang <dagang@google.com>
    
    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=245520498
    dagang authored and medb committed May 15, 2019
    Configuration menu
    Copy the full SHA
    018749f View commit details
    Browse the repository at this point in the history
  3. Initialize metadata in GoogleCloudStorageReadChannel.size() method, i…

    …f it wasn't initialized before.
    
            Change on 2019/05/13 by idv <idv@google.com>
    
    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=247978825
    medb committed May 15, 2019
    Configuration menu
    Copy the full SHA
    152ff92 View commit details
    Browse the repository at this point in the history
  4. Add property to parallelize GCS requests in listStatus and `getFile…

    …Status` methods
    
    Note: this is essentially the same change as in [] that triggered omg/12873 in the past, but it has feature flag that turns off it by default and tests that assert number of GCS requests when parallelism is enabled.
    
    In the worst case `getFileStatus` method can make up to 3 sequential requests to GCS to get implicit directory status.
    
    After moving implicit directory repair from list to delete/rename operations this worst case could be more frequent than before, because there higher chance to encounter implicit non-repaired directory:
    GoogleCloudDataproc#156
    
    This CL adds an option to execute these GCS requests in parallel which could reduce latency by up to 3 times.
    
    	Change on 2019/05/13 by idv <idv@google.com>
    
    -------------
    Created by MOE: https://github.com/google/moe
    MOE_MIGRATED_REVID=248044068
    medb committed May 15, 2019
    Configuration menu
    Copy the full SHA
    1194136 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    e6f742b View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    6fb7795 View commit details
    Browse the repository at this point in the history

Commits on May 13, 2020

  1. HWX: Modify pom to include HDP version

     Conflicts:
    	bigquery/pom.xml
    	gcs/pom.xml
    	util-hadoop/pom.xml
    	util/pom.xml
    sidseth authored and mukund-thakur committed May 13, 2020
    Configuration menu
    Copy the full SHA
    8461dc1 View commit details
    Browse the repository at this point in the history