[DEV-6356] added table installation_progress #3547

Ildyakov · 2023-04-04T18:48:16Z

This change is

Ildyakov · 2023-04-04T18:48:54Z

working with athenianco/cloud-common#138
and https://github.com/athenianco/metadata/pull/1586

lwsanty · 2023-04-04T18:55:21Z

server/athenian/api/models/state/versions/5b3dc49a9d7b_installation_progress.py

+        ),
+        sa.Column(
+            "account_created",
+            sa.TIMESTAMP(timezone=True),


can we maybe already populate this column for existing accounts? or we don't have to?
also, do we need to fill it during the account creation in accounts table?
cc @dennwc

probably we don`t need an existing accounts at all. the point of this feature is just for tracking new installations

Agree with Olek: we should create records for existing accounts. This is where the current_status will help - we can set it, but leave timestamps empty. Maybe we can update the table later to reflect actual installations that happened in the past.

Yep, we usually do this manually with a data post-migration. It's worth setting a separate task to backfill the history.

lwsanty · 2023-04-04T18:58:56Z

server/athenian/api/models/state/versions/5b3dc49a9d7b_installation_progress.py

+        sa.Column(
+            "account_id",
+            sa.Integer(),
+            sa.ForeignKey("accounts.id", name="fk_installation_progress_account"),
+            primary_key=True,
+        ),
+        sa.Column(
+            "account_created",
+            sa.TIMESTAMP(timezone=True),
+        ),
+        sa.Column(
+            "fetch_started",
+            sa.TIMESTAMP(timezone=True),
+        ),
+        sa.Column(
+            "fetch_completed",
+            sa.TIMESTAMP(timezone=True),
+        ),
+        sa.Column(
+            "consistency_started",
+            sa.TIMESTAMP(timezone=True),
+        ),
+        sa.Column(
+            "consistency_completed",
+            sa.TIMESTAMP(timezone=True),
+        ),
+        sa.Column(
+            "precompute_started",
+            sa.TIMESTAMP(timezone=True),
+        ),
+        sa.Column(
+            "precompute_completed",
+            sa.TIMESTAMP(timezone=True),
+        ),
+        sa.Column(
+            "current_status",
+            sa.Text(),
+        ),


can all these timestamps be null?

yep and they will be

server/athenian/api/models/state/models.py

dennwc · 2023-04-05T09:52:38Z

server/athenian/api/models/state/models.py

+
+    account_id = Column(
+        Integer(),
+        ForeignKey("accounts.id", name="fk_installation_progress_account"),


This FK assumes that there must be an Athenian account associated with the record. This is not the case for GitHub installations: they usually start without Athenian account. So we won't be able to insert most of the records here, unless we force Athenian account creation from MD side.

So what's your plan for the account_id column? Is it still an Athenian account ID or GitHub account ID?

this have to be a Github ID as it is in the Event

dennwc · 2023-04-05T09:54:04Z

server/athenian/api/models/state/versions/5b3dc49a9d7b_installation_progress.py

+        ),
+        sa.Column(
+            "account_created",
+            sa.TIMESTAMP(timezone=True),


Agree with Olek: we should create records for existing accounts. This is where the current_status will help - we can set it, but leave timestamps empty. Maybe we can update the table later to reflect actual installations that happened in the past.

vmarkovtsev · 2023-04-11T07:45:58Z

server/athenian/api/models/state/models.py

+
+    __tablename__ = "installation_progress"
+
+    account_id = Column(


Could you please format every field in one line for better readability and consistency with the surrounding models 🙏 E.g.

Suggested change

account_id = Column(

account_id = Column(Integer(), primary_key=True)

Also, note that primary_key=True implies nullable=False under the hood.

vmarkovtsev · 2023-04-11T07:46:52Z

server/athenian/api/models/state/models.py

+        primary_key=True,
+    )
+    account_created = Column(
+        "account_created",


We don't have to set the name for each column, luckily. This is enough:

account_created = Column(TIMESTAMP(timezone=True))

vmarkovtsev · 2023-04-11T07:47:13Z

server/athenian/api/models/state/versions/5b3dc49a9d7b_installation_progress.py

+        sa.Column(
+            "account_id",
+            sa.Integer(),
+            nullable=False,


Suggested change

nullable=False,

vmarkovtsev · 2023-04-11T07:48:19Z

server/athenian/api/models/state/versions/5b3dc49a9d7b_installation_progress.py

+        ),
+        sa.Column(
+            "account_created",
+            sa.TIMESTAMP(timezone=True),


Yep, we usually do this manually with a data post-migration. It's worth setting a separate task to backfill the history.

…loyment_labels

Bumps [morcilla[postgresql,sqlite]](https://github.com/athenianco/morcilla) from 0.5.35 to 0.5.37. - [Release notes](https://github.com/athenianco/morcilla/releases) - [Commits](athenianco/morcilla@v0.5.35...v0.5.37) --- updated-dependencies: - dependency-name: morcilla[postgresql,sqlite] dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]>

…ackage

Bumps [sentry-sdk[pure_eval]](https://github.com/getsentry/sentry-python) from 1.17.0 to 1.19.1. - [Release notes](https://github.com/getsentry/sentry-python/releases) - [Changelog](https://github.com/getsentry/sentry-python/blob/master/CHANGELOG.md) - [Commits](getsentry/sentry-python@1.17.0...1.19.1) --- updated-dependencies: - dependency-name: sentry-sdk[pure_eval] dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]>

Bumps [gcloud-aio-auth](https://github.com/talkiq/gcloud-aio) from 4.2.0 to 4.2.1. - [Release notes](https://github.com/talkiq/gcloud-aio/releases) - [Commits](talkiq/gcloud-aio@kms-4.2.0...auth-4.2.1) --- updated-dependencies: - dependency-name: gcloud-aio-auth dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]>

Bumps [gcloud-aio-pubsub](https://github.com/talkiq/gcloud-aio) from 5.3.0 to 5.4.0. - [Release notes](https://github.com/talkiq/gcloud-aio/releases) - [Commits](talkiq/gcloud-aio@pubsub-5.3.0...pubsub-5.4.0) --- updated-dependencies: - dependency-name: gcloud-aio-pubsub dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]>

Bumps [types-python-dateutil](https://github.com/python/typeshed) from 2.8.19.11 to 2.8.19.12. - [Release notes](https://github.com/python/typeshed/releases) - [Commits](https://github.com/python/typeshed/commits) --- updated-dependencies: - dependency-name: types-python-dateutil dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]>

Bumps [coverage](https://github.com/nedbat/coveragepy) from 7.2.2 to 7.2.3. - [Release notes](https://github.com/nedbat/coveragepy/releases) - [Changelog](https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst) - [Commits](nedbat/coveragepy@7.2.2...7.2.3) --- updated-dependencies: - dependency-name: coverage dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]>

Bumps [faker](https://github.com/joke2k/faker) from 18.3.1 to 18.4.0. - [Release notes](https://github.com/joke2k/faker/releases) - [Changelog](https://github.com/joke2k/faker/blob/master/CHANGELOG.md) - [Commits](joke2k/faker@v18.3.1...v18.4.0) --- updated-dependencies: - dependency-name: faker dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]>

Bumps [pytest](https://github.com/pytest-dev/pytest) from 7.2.2 to 7.3.0. - [Release notes](https://github.com/pytest-dev/pytest/releases) - [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst) - [Commits](pytest-dev/pytest@7.2.2...7.3.0) --- updated-dependencies: - dependency-name: pytest dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]>

vmarkovtsev · 2023-04-12T19:46:57Z

@Ildyakov merge conflicts, let's rebase

vmarkovtsev · 2023-04-13T09:53:40Z

I think that there is a new clean PR: #3564

[DEV-6356] added table installation_progress

b992d89

Ildyakov requested a review from lwsanty April 4, 2023 18:48

lwsanty requested a review from dennwc April 4, 2023 18:52

lwsanty reviewed Apr 4, 2023

View reviewed changes

aleksei added 2 commits April 4, 2023 14:29

[DEV-6356] update for migration

6955cab

[DEV-6356] flake update

dea89ff

dennwc suggested changes Apr 5, 2023

View reviewed changes

aleksei added 2 commits April 5, 2023 08:17

[DEV-6356] update instaprogress table

6fa6add

[DEV-6356] migration update

0da39ac

vmarkovtsev requested changes Apr 11, 2023

View reviewed changes

aleksei and others added 18 commits April 11, 2023 09:18

[DEV-6356] openapi update

48d6158

[DEV-6356] models restore

0b55fdf

[DEV-6356] fixes for migration

5634945

[DEV-6264] Unbuffered output for show_logged_query.py gcloud subproc

3f00fdd

[DEV-6264] Rename deployment_labels controllers module

375f12d

[DEV-6264] Implement internal function get_deployment_labels

d1ffa7b

[DEV-6264] Define DeploymentLabelsResponse web model

5abad5a

[DEV-6264] Implement get_deployment_labels controller

1356611

[DEV-6264] Implement internal upsert_deployment_labels and delete_dep…

a49ced2

…loyment_labels

[DEV-6264] Define DeploymentModifyLabelsRequest web model

7093240

[DEV-6264] Implement modify_deployment_labels controller

4ae7d01

[DEV-6264] Move huge test data to a separate module

c756fe4

[DEV-6264] Move a couple of misplaced tests

1768fbe

[DEV-6264] Implement invalidate_precomputed_on_labels_change()

22afbc6

[DEV-6264] Invalidate precomputed facts on deployment labels change

9e76d64

Bump version: 0.20.40 → 0.20.41

0e722cc

[DEV-6387] Add jira-acknowledge-time-below-threshold-ratio metric

0f0959c

[DEV-6387] Validate jira-ack-time threshold metric in goal update

6dd8683

gkwillie and others added 23 commits April 11, 2023 10:40

Bump version: 0.20.41 → 0.20.42

c75df89

[DEV-6403] Fix mismatching columns in extended pr deployments

dd1397c

[DEV-6368] Move invalidate_precomputed_on_labels_change() to miners p…

f447930

…ackage

[DEV-6368] Expose all_labels property in LogicalDeploymentSettings

6b4c157

[DEV-6230] Drop GraphQL endpoint and dependencies

2ab0a34

Bump version: 0.20.42 → 0.20.43

82ba63f

[DEV-6264] Fix alembic conf for rdb

fedac7e

[DEV-6264] Fix __table_args__ typo in some rdb models

f223233

Optimize _delete_deployments

4e9aa26

Bump version: 0.20.43 → 0.20.44

23ed10a

[DEV-6399] Optimize load_precomputed_done_candidates()

f26d5e9

Bump version: 0.20.44 → 0.20.45

69be81d

[DEV-6399] Solve performance bugs in query_jira_raw()

a0ab1ce

Bump version: 0.20.45 → 0.20.46

ca798c4

[DEV-6356] openapi update

4e5711e

Ildyakov requested a review from a team as a code owner April 11, 2023 15:40

vmarkovtsev closed this Apr 13, 2023

vmarkovtsev deleted the DEV-6356-migration-installation-table-create branch April 13, 2023 09:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DEV-6356] added table installation_progress #3547

[DEV-6356] added table installation_progress #3547

Ildyakov commented Apr 4, 2023 •

edited by eiso

Loading

Ildyakov commented Apr 4, 2023

lwsanty Apr 4, 2023

Ildyakov Apr 4, 2023

dennwc Apr 5, 2023

vmarkovtsev Apr 11, 2023

lwsanty Apr 4, 2023

Ildyakov Apr 4, 2023

dennwc Apr 5, 2023

Ildyakov Apr 5, 2023

dennwc Apr 6, 2023

Ildyakov Apr 7, 2023

dennwc Apr 5, 2023

vmarkovtsev Apr 11, 2023

vmarkovtsev Apr 11, 2023

vmarkovtsev Apr 11, 2023

vmarkovtsev Apr 11, 2023

vmarkovtsev commented Apr 12, 2023 •

edited

Loading

vmarkovtsev commented Apr 13, 2023

	account_id = Column(
	account_id = Column(Integer(), primary_key=True)

[DEV-6356] added table installation_progress #3547

[DEV-6356] added table installation_progress #3547

Conversation

Ildyakov commented Apr 4, 2023 • edited by eiso Loading

Ildyakov commented Apr 4, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vmarkovtsev commented Apr 12, 2023 • edited Loading

vmarkovtsev commented Apr 13, 2023

Ildyakov commented Apr 4, 2023 •

edited by eiso

Loading

vmarkovtsev commented Apr 12, 2023 •

edited

Loading