Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DEV-6356] added table installation_progress #3547

Closed
wants to merge 46 commits into from
Closed
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
b992d89
[DEV-6356] added table installation_progress
Apr 4, 2023
6955cab
[DEV-6356] update for migration
Apr 4, 2023
dea89ff
[DEV-6356] flake update
Apr 4, 2023
6fa6add
[DEV-6356] update instaprogress table
Apr 5, 2023
0da39ac
[DEV-6356] migration update
Apr 5, 2023
48d6158
[DEV-6356] openapi update
Apr 11, 2023
0b55fdf
[DEV-6356] models restore
Apr 11, 2023
5634945
[DEV-6356] fixes for migration
Apr 11, 2023
3f00fdd
[DEV-6264] Unbuffered output for show_logged_query.py gcloud subproc
Apr 4, 2023
375f12d
[DEV-6264] Rename deployment_labels controllers module
Apr 3, 2023
d1ffa7b
[DEV-6264] Implement internal function get_deployment_labels
Apr 3, 2023
5abad5a
[DEV-6264] Define DeploymentLabelsResponse web model
Apr 3, 2023
1356611
[DEV-6264] Implement get_deployment_labels controller
Apr 3, 2023
a49ced2
[DEV-6264] Implement internal upsert_deployment_labels and delete_dep…
Apr 3, 2023
7093240
[DEV-6264] Define DeploymentModifyLabelsRequest web model
Apr 3, 2023
4ae7d01
[DEV-6264] Implement modify_deployment_labels controller
Apr 4, 2023
c756fe4
[DEV-6264] Move huge test data to a separate module
Apr 4, 2023
1768fbe
[DEV-6264] Move a couple of misplaced tests
Apr 4, 2023
22afbc6
[DEV-6264] Implement invalidate_precomputed_on_labels_change()
Apr 5, 2023
9e76d64
[DEV-6264] Invalidate precomputed facts on deployment labels change
Apr 5, 2023
0e722cc
Bump version: 0.20.40 → 0.20.41
gkwillie Apr 5, 2023
0f0959c
[DEV-6387] Add jira-acknowledge-time-below-threshold-ratio metric
Apr 7, 2023
6dd8683
[DEV-6387] Validate jira-ack-time threshold metric in goal update
Apr 7, 2023
c75df89
Bump version: 0.20.41 → 0.20.42
gkwillie Apr 7, 2023
9eb9962
Bump morcilla[postgresql,sqlite] from 0.5.35 to 0.5.37 in /server
dependabot[bot] Apr 3, 2023
dd1397c
[DEV-6403] Fix mismatching columns in extended pr deployments
vmarkovtsev Apr 10, 2023
f447930
[DEV-6368] Move invalidate_precomputed_on_labels_change() to miners p…
Apr 6, 2023
6b4c157
[DEV-6368] Expose all_labels property in LogicalDeploymentSettings
Apr 6, 2023
2ab0a34
[DEV-6230] Drop GraphQL endpoint and dependencies
vmarkovtsev Apr 10, 2023
82ba63f
Bump version: 0.20.42 → 0.20.43
gkwillie Apr 10, 2023
613557c
Bump sentry-sdk[pure_eval] from 1.17.0 to 1.19.1 in /server
dependabot[bot] Apr 10, 2023
fedac7e
[DEV-6264] Fix alembic conf for rdb
Apr 4, 2023
f223233
[DEV-6264] Fix __table_args__ typo in some rdb models
Apr 4, 2023
4e9aa26
Optimize _delete_deployments
vmarkovtsev Apr 10, 2023
acc5dff
Bump gcloud-aio-auth from 4.2.0 to 4.2.1 in /server
dependabot[bot] Apr 10, 2023
23ed10a
Bump version: 0.20.43 → 0.20.44
gkwillie Apr 10, 2023
30fbb69
Bump gcloud-aio-pubsub from 5.3.0 to 5.4.0 in /server
dependabot[bot] Apr 10, 2023
117ec37
Bump types-python-dateutil from 2.8.19.11 to 2.8.19.12 in /server
dependabot[bot] Apr 10, 2023
de9e9fe
Bump coverage from 7.2.2 to 7.2.3 in /server
dependabot[bot] Apr 10, 2023
fbac3b0
Bump faker from 18.3.1 to 18.4.0 in /server
dependabot[bot] Apr 10, 2023
f26d5e9
[DEV-6399] Optimize load_precomputed_done_candidates()
vmarkovtsev Apr 11, 2023
69be81d
Bump version: 0.20.44 → 0.20.45
gkwillie Apr 11, 2023
a4b2d31
Bump pytest from 7.2.2 to 7.3.0 in /server
dependabot[bot] Apr 11, 2023
a0ab1ce
[DEV-6399] Solve performance bugs in query_jira_raw()
vmarkovtsev Apr 11, 2023
ca798c4
Bump version: 0.20.45 → 0.20.46
gkwillie Apr 11, 2023
4e5711e
[DEV-6356] openapi update
Apr 11, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 46 additions & 1 deletion server/athenian/api/models/state/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -595,7 +595,7 @@ class DashboardChart(create_time_mixin(created_at=True, updated_at=True), Base):


class DashboardChartGroupBy(create_time_mixin(created_at=True, updated_at=True), Base):
"""The group by configured for a dashboard chart."""
"""Tracking installation progress for github accounts"""
dennwc marked this conversation as resolved.
Show resolved Hide resolved

__tablename__ = "dashboard_charts_group_by"

Expand Down Expand Up @@ -640,3 +640,48 @@ class DashboardChartGroupBy(create_time_mixin(created_at=True, updated_at=True),
DashboardChartGroupBy.jira_issue_types,
DashboardChartGroupBy.jira_labels,
)


class InstallationProgress(create_time_mixin(created_at=False, updated_at=False), Base):
"""The group by configured for a dashboard chart."""

__tablename__ = "installation_progress"

account_id = Column(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please format every field in one line for better readability and consistency with the surrounding models 🙏 E.g.

Suggested change
account_id = Column(
account_id = Column(Integer(), primary_key=True)

Also, note that primary_key=True implies nullable=False under the hood.

Integer(),
ForeignKey("accounts.id", name="fk_installation_progress_account"),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This FK assumes that there must be an Athenian account associated with the record. This is not the case for GitHub installations: they usually start without Athenian account. So we won't be able to insert most of the records here, unless we force Athenian account creation from MD side.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok removed

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So what's your plan for the account_id column? Is it still an Athenian account ID or GitHub account ID?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this have to be a Github ID as it is in the Event

nullable=False,
primary_key=True,
)
account_created = Column(
"account_created",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't have to set the name for each column, luckily. This is enough:

account_created = Column(TIMESTAMP(timezone=True))

TIMESTAMP(timezone=True),
)
fetch_started = Column(
"fetch_started",
TIMESTAMP(timezone=True),
)
fetch_completed = Column(
"fetch_completed",
TIMESTAMP(timezone=True),
)
consistency_started = Column(
"consistency_started",
TIMESTAMP(timezone=True),
)
consistency_completed = Column(
"consistency_completed",
TIMESTAMP(timezone=True),
)
precompute_started = Column(
"precompute_started",
TIMESTAMP(timezone=True),
)
precompute_completed = Column(
"precompute_completed",
TIMESTAMP(timezone=True),
)
current_status = Column(
"current_status",
Text(),
)
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
"""Installation progress

Revision ID: 5b3dc49a9d7b
Revises: 3557bf5f319a
Create Date: 2023-04-04 17:13:06.137518+00:00

"""
from alembic import op
import sqlalchemy as sa

# revision identifiers, used by Alembic.
revision = "5b3dc49a9d7b"
down_revision = "3557bf5f319a"
branch_labels = None
depends_on = None


def upgrade():
op.create_table(
"installation_progress",
sa.Column(
"account_id",
sa.Integer(),
sa.ForeignKey("accounts.id", name="fk_installation_progress_account"),
nullable=False,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
nullable=False,

primary_key=True,
),
sa.Column(
"account_created",
sa.TIMESTAMP(timezone=True),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we maybe already populate this column for existing accounts? or we don't have to?
also, do we need to fill it during the account creation in accounts table?
cc @dennwc

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably we don`t need an existing accounts at all. the point of this feature is just for tracking new installations

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree with Olek: we should create records for existing accounts. This is where the current_status will help - we can set it, but leave timestamps empty. Maybe we can update the table later to reflect actual installations that happened in the past.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, we usually do this manually with a data post-migration. It's worth setting a separate task to backfill the history.

),
sa.Column(
"fetch_started",
sa.TIMESTAMP(timezone=True),
),
sa.Column(
"fetch_completed",
sa.TIMESTAMP(timezone=True),
),
sa.Column(
"consistency_started",
sa.TIMESTAMP(timezone=True),
),
sa.Column(
"consistency_completed",
sa.TIMESTAMP(timezone=True),
),
sa.Column(
"precompute_started",
sa.TIMESTAMP(timezone=True),
),
sa.Column(
"precompute_completed",
sa.TIMESTAMP(timezone=True),
),
sa.Column(
"current_status",
sa.Text(),
),
)


def downgrade():
op.drop_table("installation_progress")