Old databases migrations #1038

vlejd · 2017-04-01T12:52:34Z

Migration scripts for kms, fks, ksp and some csv files.

We do not migrate old competitions nor old camps.

codecov · 2017-04-01T12:53:07Z

Codecov Report

Merging #1038 into master will decrease coverage by 3.13%.
The diff coverage is 4.27%.

@@            Coverage Diff            @@
##           master   #1038      +/-   ##
=========================================
- Coverage   70.13%     67%   -3.14%     
=========================================
  Files         107     112       +5     
  Lines        5126    5388     +262     
  Branches      672     707      +35     
=========================================
+ Hits         3595    3610      +15     
- Misses       1417    1664     +247     
  Partials      114     114

Impacted Files	Coverage Δ
trojsten/people/helpers.py	`95.91% <ø> (ø)`	⬆️
...n/people/management/commands/migrate_ksp_kaspar.py	`0% <0%> (ø)`
...ple/management/commands/migrate_30rokovfks1_csv.py	`0% <0%> (ø)`
...ple/management/commands/migrate_30rokovfks2_csv.py	`0% <0%> (ø)`
...sten/people/management/commands/migrate_kms_csv.py	`0% <0%> (ø)`
...n/people/management/commands/migrate_base_class.py	`0% <0%> (ø)`
...sten/people/management/commands/migrate_fks_csv.py	`0% <0%> (ø)`
trojsten/people/constants.py	`100% <100%> (ø)`	⬆️
... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update db5808e...df2bdbd. Read the comment docs.

mhozza · 2017-04-01T13:22:52Z

Preco su tam 2 fks migracne skripty? aky je medzi nimi rozdiel?

Predpokladam ze kedze to pouzijeme asi len raz, tak sa nam k tomu asi nechce pisat testy, ale v tom pripade si chceme byt isty ze to robi co ma a pouzit to skor ako to bude out of date.

btw treba to zrejme rebasnut.

vlejd · 2017-04-02T09:02:58Z

Su tam 3 FKS skripty. Jeden je na staru databazu, a dva su na dva rvozne druhy CSV ciek z akcie 30 rokov fks.

Rad k tomu napisem nejake testy, len si niesom uplne isty ako. Mockovat vsetky tie databazy a subory by som nerad. Mozem ale napisat checker na niekolko userov a to, ako sa mali pomigrovat.

vlejd · 2017-06-17T21:16:05Z

@mhozza @maaario kukne sa mi na to uz niekto?

mhozza

Sorry ze to tolko trvalo, ale tu mas.
Zatial som pozerak ksp a base.
Skus aplikovat pripomienky aj na ostatne veci, potom to pozriem cele.

mhozza · 2017-06-23T19:16:44Z

trojsten/people/management/commands/migrate_ksp_kaspar.py

+BIRTHDAY_PROP = 2
+
+
+class Command(MigrateBaceCommand):


mhozza · 2017-06-23T19:19:09Z

trojsten/people/management/commands/migrate_ksp_kaspar.py

+        for row in c:
+            self.process_school(*row)
+
+        # TODO sustredka


Mozno by bolo dobre tuna linknut issue na githube, aby to nebol len nahodny comment v kode, na ktory sa zabudne.

mhozza · 2017-06-23T19:20:00Z

trojsten/people/management/commands/migrate_ksp_kaspar.py

+class Command(MigrateBaceCommand):
+    help = 'Imports people and their related info from kaspar.'
+
+    def handle_noargs(self, **options):


Tato metoda toho robi uz prilis vela, treba to rozdelit do mensich metod.

ziskanie zoznamu sustrediek

ziskanie zoznamu ucastnikov

ziskanie poctu sustredeni

mhozza · 2017-06-23T19:45:16Z

trojsten/people/management/commands/migrate_base_class.py

+sys.setdefaultencoding("utf-8")
+
+
+class MigrateBaceCommand(NoArgsCommand):


mhozza · 2017-06-23T19:46:14Z

trojsten/people/management/commands/migrate_base_class.py

+                            dest='dry',
+                            default=True,
+                            help='Actually write something to DB')
+        parser.add_argument('--fast',


Toto chce lepsi description - co je to "few users"?

mhozza · 2017-06-23T20:39:46Z

trojsten/people/management/commands/migrate_base_class.py

+        if self.dry:
+            new_user = User(**user_args)
+        else:
+            addr = None


naco je tu tato premenna - ak dobre vidim, addr sa pouzije len na riadku 197 a medzitym je aj tak prepisana.

mhozza · 2017-06-23T20:40:15Z

trojsten/people/management/commands/migrate_base_class.py

+                                            address['town'],
+                                            address['postal_code'],
+                                            address['country'])
+                user_args['home_address'] = addr


tu mozes rovno priradit to co priradujes do addr a addr vyhodit

mhozza · 2017-06-23T20:43:26Z

trojsten/people/management/commands/migrate_base_class.py

+                if valid_contacts:
+                    user_properties.append([self.LAST_CONTACT_PROPERTY, max(valid_contacts)])
+
+            user_properties = list(filter(lambda x: x, user_properties))


Toto je velmi obskurne. Co napriklad:
user_properties = [prop for prop in user_properties if prop is not None]

mhozza · 2017-06-23T20:45:04Z

trojsten/people/management/commands/migrate_base_class.py

+
+        similar_users = get_similar_users(new_user)
+        if len(similar_users):
+            names_of_similar = [(x.first_name, x.last_name) for x in similar_users]


Nie som uplne fanusik pouzivania premennej x na vsetko ale whatever.

mhozza · 2017-06-23T20:58:52Z

trojsten/people/management/commands/migrate_base_class.py

+            self.similar_users.append(((first_name, last_name), names_of_similar))
+            if self.verbosity >= 2:
+                self.stdout.write('Similar users: %s' % str(names_of_similar))
+            if self.dry:


co napr. if not self.dry:

matejbadin · 2017-08-14T13:36:11Z

@vlejd @mhozza Ahojte, v akom je toto momentálne stave?

mhozza · 2017-08-16T10:31:07Z

Pokial mi je zname, od mojho podledneho review sa nic neudialo (Vlejd tusim pise diplomovku alebo co)

vlejd · 2017-08-21T17:46:55Z

Momentalny stav je, ze si vybavujem povolenie v googli, aby som mohol nico urobit. Ono google ma narok na vsetok kod ktory napises ked si tam zamestnany a mohli by z toho byt problemy.

mhozza · 2017-08-21T19:02:24Z

Ja som to nejak neriešil.

…

On Mon, Aug 21, 2017, 18:46 vlejd ***@***.***> wrote: Momentalny stav je, ze si vybavujem povolenie v googli, aby som mohol nico urobit. Ono google ma narok na vsetok kod ktory napises ked si tam zamestnany a mohli by z toho byt problemy. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1038 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABH4N629Wqp98Ds32cUkm8knGimY992cks5sacKQgaJpZM4Mwepd> .

…r-migration

mhozza · 2017-11-28T17:23:00Z

trojsten/people/management/commands/migrate_base_class.py

@@ -120,7 +121,8 @@ def create_school(self, abbr, name, addr_name, street,
        abbr += '?'  # Question mark denotes schools needing review.
        school = None
        if len(zip_code) > 10:
-            # Swiss zip codes
+            # Swiss zip codes are longer than 10 chars, but our db model does not allow


Nechceme zmenit model aby ich podporoval?
Ak sa nam nechce teraz, mozes sem pridat TODO

Ake dlhe su svajciarske PSC? Podla wikipedia a google map 4 miesta.

trojsten/people/management/commands/migrate_base_class.py

black3r · 2018-03-11T09:35:22Z

V python 3 nefunguje, skonci s chybou AttributeError: module 'sys' has no attribute 'setdefaultencoding'

black3r · 2018-03-11T10:03:03Z

Po lokálnom spustení s python2
python manage.py migrate_fks_csv data/csv/fks/ --wet_run skonci s chybou

Traceback (most recent call last):
  File "manage.py", line 40, in <module>
    execute_from_command_line(sys.argv)
  File "/home/roman/.pyenv/versions/trojsten-2/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/home/roman/.pyenv/versions/trojsten-2/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/roman/.pyenv/versions/trojsten-2/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/roman/.pyenv/versions/trojsten-2/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/home/roman/work/trojsten/web/trojsten/people/management/commands/migrate_fks_csv.py", line 100, in handle
    address=parsed_address)
  File "/home/roman/.pyenv/versions/trojsten-2/lib/python2.7/site-packages/django/utils/decorators.py", line 185, in inner
    return func(*args, **kwargs)
  File "/home/roman/work/trojsten/web/trojsten/people/management/commands/migrate_base_class.py", line 200, in process_person
    postal_code=address['postal_code'], country=address['country'])
  File "/home/roman/.pyenv/versions/trojsten-2/lib/python2.7/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/roman/.pyenv/versions/trojsten-2/lib/python2.7/site-packages/django/db/models/query.py", line 394, in create
    obj.save(force_insert=True, using=self.db)
  File "/home/roman/.pyenv/versions/trojsten-2/lib/python2.7/site-packages/django/db/models/base.py", line 808, in save
    force_update=force_update, update_fields=update_fields)
  File "/home/roman/.pyenv/versions/trojsten-2/lib/python2.7/site-packages/django/db/models/base.py", line 838, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/home/roman/.pyenv/versions/trojsten-2/lib/python2.7/site-packages/django/db/models/base.py", line 924, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/home/roman/.pyenv/versions/trojsten-2/lib/python2.7/site-packages/django/db/models/base.py", line 963, in _do_insert
    using=using, raw=raw)
  File "/home/roman/.pyenv/versions/trojsten-2/lib/python2.7/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/roman/.pyenv/versions/trojsten-2/lib/python2.7/site-packages/django/db/models/query.py", line 1076, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/home/roman/.pyenv/versions/trojsten-2/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 1112, in execute_sql
    cursor.execute(sql, params)
  File "/home/roman/.pyenv/versions/trojsten-2/lib/python2.7/site-packages/django/db/backends/utils.py", line 79, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "/home/roman/.pyenv/versions/trojsten-2/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/home/roman/.pyenv/versions/trojsten-2/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/home/roman/.pyenv/versions/trojsten-2/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
django.db.utils.DataError: value too long for type character varying(2)

mhozza · 2019-07-09T20:33:05Z

Uz by to malo bezat pod py3.
@black3r Ako si to skusal lokalne? odkial mas data na migraciu?

…iable reuse.

vlejd added 6 commits March 22, 2017 22:08

Revised kaspar migrating script.

8ae2ac9

kms migration skript + improved mihration structure

1ed292a

Good enough migration scripts.

e399f1f

Whole pipeline is working on wet run.

b16f898

pep-8 fixes

c45a524

Fixed lint errors and added some options

fd387f8

vlejd requested review from mhozza, Zajozor and matejbadin April 1, 2017 12:52

Minor parameter changes.

3937866

vlejd requested a review from maaario June 17, 2017 21:09

mhozza requested changes Jun 23, 2017

View reviewed changes

vlejd added 10 commits October 22, 2017 14:09

Revised kaspar migrating script.

1f62905

kms migration skript + improved mihration structure

dcd9d35

Good enough migration scripts.

d149519

Whole pipeline is working on wet run.

e7e472e

pep-8 fixes

f4f4039

Fixed lint errors and added some options

dc73736

Minor parameter changes.

adffa56

Merge branch 'kaspar-migration' of github.com:trojsten/web into kaspa…

fd07ac2

…r-migration

Merge branch 'master' into kaspar-migration

f362320

Review fixes part 1.

b176aa0

mhozza approved these changes Nov 28, 2017

View reviewed changes

mhozza added 4 commits July 9, 2019 19:20

Fix typo.

f3d7fb6

Merge branch 'master' of github.com:trojsten/web into kaspar-migration

8e08386

black and isort.

4094bda

Fix some review comments.

48944c6

mhozza added 3 commits July 9, 2019 21:36

Use raw string literals for regexps.

e476954

Use raw string literals for strings containing regexps and ignore var…

0320953

…iable reuse.

Merge branch 'master' into kaspar-migration

df2bdbd

mhozza requested a review from black3r July 14, 2019 18:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Old databases migrations #1038

Old databases migrations #1038

vlejd commented Apr 1, 2017

codecov bot commented Apr 1, 2017 •

edited

Loading

mhozza commented Apr 1, 2017

vlejd commented Apr 2, 2017

vlejd commented Jun 17, 2017

mhozza left a comment

mhozza Jun 23, 2017

mhozza Jun 23, 2017

mhozza Jun 23, 2017

vlejd Oct 30, 2017

mhozza Jun 23, 2017

vlejd Oct 30, 2017

mhozza Jun 23, 2017

vlejd Oct 30, 2017

mhozza Jun 23, 2017

vlejd Oct 30, 2017

mhozza Jun 23, 2017

vlejd Oct 30, 2017

mhozza Jun 23, 2017

vlejd Oct 30, 2017

mhozza Jun 23, 2017

vlejd Oct 30, 2017

mhozza Jun 23, 2017

vlejd Oct 30, 2017

matejbadin commented Aug 14, 2017

mhozza commented Aug 16, 2017

vlejd commented Aug 21, 2017

mhozza commented Aug 21, 2017 via email

mhozza Nov 28, 2017

mhozza Jul 9, 2019

black3r commented Mar 11, 2018

black3r commented Mar 11, 2018 •

edited by maaario

Loading

mhozza commented Jul 9, 2019

		sys.setdefaultencoding("utf-8")


		class MigrateBaceCommand(NoArgsCommand):

Old databases migrations #1038

Are you sure you want to change the base?

Old databases migrations #1038

Conversation

vlejd commented Apr 1, 2017

codecov bot commented Apr 1, 2017 • edited Loading

Codecov Report

mhozza commented Apr 1, 2017

vlejd commented Apr 2, 2017

vlejd commented Jun 17, 2017

mhozza left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matejbadin commented Aug 14, 2017

mhozza commented Aug 16, 2017

vlejd commented Aug 21, 2017

mhozza commented Aug 21, 2017 via email

Choose a reason for hiding this comment

Choose a reason for hiding this comment

black3r commented Mar 11, 2018

black3r commented Mar 11, 2018 • edited by maaario Loading

mhozza commented Jul 9, 2019

codecov bot commented Apr 1, 2017 •

edited

Loading

black3r commented Mar 11, 2018 •

edited by maaario

Loading