GitLab
This page contains the setup guide and reference information for the Gitlab Source connector.
Prerequisites
- Gitlab instance or an account at Gitlab
For Airbyte Cloud:
- Personal Access Token (see personal access token)
- OAuth
For Airbyte Open Source:
- Personal Access Token (see personal access token)
Setup guide
Step 1: Set up GitLab
Create a GitLab Account or set up a local instance of GitLab.
Airbyte Open Source additional setup steps
Log into GitLab and then generate a personal access token. Your token should have the read_api scope, that Grants read access to the API, including all groups and projects, the container registry, and the package registry.
Step 2: Set up the GitLab connector in Airbyte
For Airbyte Cloud:
- Log into your Airbyte Cloud account.
- In the left navigation bar, click Sources. In the top-right corner, click + new source.
- On the source setup page, select GitLab from the Source type dropdown and enter a name for this connector.
- Click
Authenticate your GitLab accountby selecting Oauth or Personal Access Token for Authentication. - Log in and Authorize to the GitLab account.
- API URL (Optional) - The URL to access your self-hosted GitLab instance or
gitlab.com(default). - Start date (Optional) - The date from which you'd like to replicate data for streams.
- Groups (Optional) - List of GitLab group IDs, e.g.
airbytehqfor single group,airbytehq another-repofor multiple groups. - Projects (Optional) - List of GitLab projects to pull data for, e.g.
airbytehq/airbyte. - Click Set up source.
Note: You can specify either Group IDs or Project IDs in the source configuration. If both fields are blank, the connector will retrieve a list of all the groups that are accessible to the configured token and ingest as normal.
For Airbyte Open Source:
- Authenticate with Personal Access Token.
Supported sync modes
The Gitlab Source connector supports the following sync modes:
Supported Streams
This connector outputs the following streams:
- Branches
- Commits (Incremental)
- Issues (Incremental)
- Group Issue Boards
- Pipelines (Incremental)
- Jobs
- Projects
- Project Milestones
- Project Merge Requests (Incremental)
- Users
- Groups
- Group Milestones
- Group and Project members
- Tags
- Releases
- Deployments
- Group Labels
- Project Labels
- Epics (only available for GitLab Ultimate and GitLab.com Gold accounts. Stream Epics uses iid field as primary key for more convenient search and matching with UI. Iid is the internal ID of the epic, number of Epic on UI.)
- Epic Issues (only available for GitLab Ultimate and GitLab.com Gold accounts)
Additional information
GitLab source works with GitLab API v4. It can also work with self-hosted GitLab API v4.
Performance considerations
Gitlab has the rate limits, but the Gitlab connector should not run into Gitlab API limitations under normal usage. Please create an issue if you see any rate limit issues that are not automatically retried successfully.
Reference
Config fields reference
Field
Type
Property name
object
credentials
string
start_date
string
api_url
string
groups
array<string>
groups_list
string
projects
array<string>
projects_list
Changelog
Expand to review
| Version | Date | Pull Request | Subject |
|---|---|---|---|
| 4.0.5 | 2024-07-06 | 40894 | Update dependencies |
| 4.0.4 | 2024-06-25 | 40417 | Update dependencies |
| 4.0.3 | 2024-06-22 | 40102 | Update dependencies |
| 4.0.2 | 2024-04-24 | 36637 | Schema descriptions and CDK 0.80.0 |
| 4.0.1 | 2024-04-23 | 37505 | Set error code 500 as retryable |
| 4.0.0 | 2024-03-25 | 35989 | Migrate to low-code |
| 3.0.0 | 2024-01-25 | 34548 | Fix merge_request_commits stream to return commits for each merge request |
| 2.1.2 | 2024-02-12 | 35167 | Manage dependencies with Poetry. |
| 2.1.1 | 2024-01-12 | 34203 | prepare for airbyte-lib |
| 2.1.0 | 2023-12-20 | 33676 | Add fields to Commits (extended_trailers), Groups (emails_enabled, service_access_tokens_expiration_enforced) and Projects (code_suggestions, model_registry_access_level) streams |
| 2.0.0 | 2023-10-23 | 31700 | Add correct date-time format for Deployments, Projects and Groups Members streams |
| 1.8.4 | 2023-10-19 | 31599 | Base image migration: remove Dockerfile and use the python-connector-base image |
| 1.8.3 | 2023-10-18 | 31547 | Add validation for invalid groups_list and/or projects_list |
| 1.8.2 | 2023-10-17 | 31492 | Expand list of possible error status codes when handling expired access_token |
| 1.8.1 | 2023-10-12 | 31375 | Mark start_date as optional, migrate groups and projects to array |
| 1.8.0 | 2023-10-12 | 31339 | Add undeclared fields to streams schemas, validate date/date-time format in stream schemas |
| 1.7.1 | 2023-10-10 | 31210 | Added expired access_token handling, while checking the connection |
| 1.7.0 | 2023-08-08 | 27869 | Add Deployments stream |
| 1.6.0 | 2023-06-30 | 27869 | Add shared_runners_setting field to groups |
| 1.5.1 | 2023-06-24 | 27679 | Fix formatting |
| 1.5.0 | 2023-06-15 | 27392 | Make API URL an optional parameter in spec. |
| 1.4.2 | 2023-06-15 | 27346 | Partially revert changes made in version 1.0.4, disallow http calls in cloud. |
| 1.4.1 | 2023-06-13 | 27351 | Fix OAuth token expiry date. |
| 1.4.0 | 2023-06-12 | 27234 | Skip stream slices on 403/404 errors, do not fail syncs. |
| 1.3.1 | 2023-06-08 | 27147 | Improve connectivity check for connections with no projects/groups |
| 1.3.0 | 2023-06-08 | 27150 | Update stream schemas |
| 1.2.1 | 2023-06-02 | 26947 | New field name added to Pipelines and PipelinesExtended stream schema |
| 1.2.0 | 2023-05-17 | 22293 | Preserve data in records with flattened keys |
| 1.1.1 | 2023-05-23 | 26422 | Fix error 404 Repository Not Found when syncing project with Repository feature disabled |
| 1.1.0 | 2023-05-10 | 25948 | Introduce two new fields in the Projects stream schema |
| 1.0.4 | 2023-04-20 | 21373 | Accept api_url with or without scheme |
| 1.0.3 | 2023-02-14 | 22992 | Specified date formatting in specification |
| 1.0.2 | 2023-01-27 | 22001 | Set AvailabilityStrategy for streams explicitly to None |
| 1.0.1 | 2023-01-23 | 21713 | Fix missing data issue |
| 1.0.0 | 2022-12-05 | 7506 | Add OAuth2.0 authentication option |
| 0.1.12 | 2022-12-15 | 20542 | Revert HttpAvailability changes, run on cdk 0.15.0 |
| 0.1.11 | 2022-12-14 | 20479 | Use HttpAvailabilityStrategy + add unit tests |
| 0.1.10 | 2022-12-12 | 20384 | Fetch groups along with their subgroups |
| 0.1.9 | 2022-12-11 | 20348 | Fix 403 error when syncing EpicIssues stream |
| 0.1.8 | 2022-12-02 | 20023 | Fix duplicated records issue for Projects stream |
| 0.1.7 | 2022-12-01 | 19986 | Fix GroupMilestones stream schema |
| 0.1.6 | 2022-06-23 | 13252 | Add GroupIssueBoards stream |
| 0.1.5 | 2022-05-02 | 11907 | Fix null projects param and container_expiration_policy |
| 0.1.4 | 2022-03-23 | 11140 | Ingest All Accessible Groups if not Specified in Config |
| 0.1.3 | 2021-12-21 | 8991 | Update connector fields title/description |
| 0.1.2 | 2021-10-18 | 7108 | Allow all domains to be used as api_url |
| 0.1.1 | 2021-10-12 | 6932 | Fix pattern field in spec file, remove unused fields from config files, use cache from CDK |
| 0.1.0 | 2021-07-06 | 4174 | Initial Release |