mirror/peeringdb-peeringdb
1
0
Fork 0
mirror of https://github.com/peeringdb/peeringdb.git synced 2024-05-11 05:55:09 +00:00
Code Releases Activity

July updates (#762)

Browse Source 
* Change label from primary ASN to ASN

* Raise validation error when trying to update ASN

* first steps for dotf importer procotol (#697)

* migrations (#697)

* Add translation to error meessage

* Make ASN readonly in table

* Add test now that ASN should not be able to update

* Set fac.rencode to '' for all entries and make it readonly in serializer

* Add unique constraints to network ixlan ip addresses

* Add migration to null out duplicate ipaddresses for deleted netixlans

* Add unique constraints to network ixlan ip addresses

* Add migration to null out duplicate ipaddresses for deleted netixlans

* remove old migrations (#697)

* fix netixlan ipaddr dedupe migration (#268)
add netixlan ipaddr unique constraint migration (#268)

* ixf_member_data migrations (#697)

* fix table name (#697)

* importer protocol (#697)

* fix netixlan ipaddr dedupe migration (#268)
add netixlan ipaddr unique constraint migration (#268)

* ixf proposed changes notifications (#697)

* Delete repeated query

* Add a test to show rencode is readonly

* Blank out rencode when mocking data

* Remove validator now that constraint exists

* Add back unique field validator w Check Deleted true

* conflict resolving (#697)

* UniqueFieldValidator raise error with code "unique" (#268)

* conflict resolution (#697)

* Add fixme comment to tests

* conflict resolution (#697)

* Remove now invalid undelete tests

* UniqueFieldValidator raise error with code "unique" (#268)

* delete admin tools for duplicate ip addresses

* Make migration to delete duplicateipnetworkixlan

* Add ixlan-ixpfx status matching validation, add corresponding test

* delete redundant checking in test

* resolve conflict ui (#697)

* fix migrations hierarchy

* squash migrations for ixf member data

* clean up preview and post-mortem tools

* remove non-sensical permission check when undeleting soft-deleted objects through unique integrity error handling

* only include the ix-f data url in notifications to admincom (#697)

* resolve on --skip-import (#697)

* ac conflict resolution (#697)

* Define more accurately the incompatible statuses for ixlan and ixpfx

* Add another status test

* Preventing disrupting changes (#697)

* fix tests (#697)

* Stop allow_ixp_update from being write only and add a global stat for automated networks

* Add tests for global stats that appear in footer

* Change how timezone is called with datetime, to get test_stats.py/test_generate_for_current_date to pass

* test for protected entities (#697)

* admincom conflict resolution refine readonly fields (#697)
network notifications only if the problem is actually actionable by the network (#697)

* ixp / ac notifcation when ix-f source cannot be parsed (#697)
fix issue with ixlan prefix protection (#697)

* migrations (#697)

* code documentation (#697)

* ux tweaks (#697)

* UX tweaks (#697)

* Fix typo

* fix netixlan returned in IXFMemberData.apply when adding a new one (#697)

* fix import log incosistencies (#697)

* Add IXFMemberData to test

* Update test data

* Add protocol tests

* Add tests for views

* always persist changes to remote data on set_conflict (#697)

* More tests

* always persist changes to remote data on set_conflict (#697)

* suggest-add test

* net_present_at_ix should check status (#697)

* Add more protocol tests

* Edit language of some tests

* django-peeringdb to 2.1.1
relock pipfile, pin django-ratelimit to <3 as it breaks stuff

* Add net_count_ixf field to ix object (#683)

* Add the IX-F Member Export URL to the ixlan API endpoint (#249)

* Lock some objects from being deleted by the owner (#696)

* regenerate api docs (#249)

* always persist changes to remote data on set_add and set_update (#697)

* IXFMemberData: always persist remote data changes during set_add and set_update, also allow for saving without touching the updated field

* always persist changes to remote data on set_add and set_update (#697)

* Fix suggest-add tests

* IXFMemberData: always persist remote data changes during set_add and set_update, also allow for saving without touching the updated field

* IXFMemberData: always persist remote data changes during set_add and set_update, also allow for saving without touching the updated field

* fix issue with deletion when ixfmemberdata for entry existed previously (#697)

* fix test_suggest_delete_local_ixf_no_flag (#697 tests)

* fix issue with deletion when ixfmemberdata for entry existed previously (#697)

* invalid ips get logged and notified to the ix via notify_error (#697)

* Fix more tests

* issue with previous_data when running without save (#697)
properly track speed errors (#697)

* reset errors on ixfmemberdata that go into pending_save (#697)

* add remote_data to admin view (#697)

* fix error reset inconsistency (#697)

* Refine invalid data tests

* remove debug output

* for notifications to ac include contact points for net and ix in the message (#697)

* settings to toggle ix-f tickets / emails (#697)

* allow turning off ix-f notifications for net and ix separately (#697)

* add jsonschema test

* Add idempotent tests to updater

* remove old ixf member tests

* Invalid data tests when ixp_updates are enabled

* fix speed error validation (#697)

* fix issue with rollback (#697)

* fix migration hierarchy

* fix ixfmemberdata _email

* django-peeringdb to 2.2 and relock

* add ixf rollback tests

* ixf email notifications off by default

* black formatted

* pyupgrade

Co-authored-by: egfrank <egfrank@20c.com>
Co-authored-by: Stefan Pratter <stefan@20c.com>
This commit is contained in:
Matt Griswold
2020-07-15 02:07:01 -05:00
committed by GitHub
parent 462c92a288
commit ea55c4dc38
108 changed files with 8908 additions and 2541 deletions
Download Patch File Download Diff File Expand all files Collapse all files

248
peeringdb_server/management/commands/pdb_api_test.py
View File

@@ -47,6 +47,7 @@ from peeringdb_server.models import (
IXLan,
IXLanPrefix,
InternetExchangeFacility,
DeskProTicket,
)
from peeringdb_server.serializers import REFTAG_MAP as REFTAG_MAP_SLZ
@@ -284,7 +285,7 @@ class TestJSON(unittest.TestCase):
"zipcode": ZIPCODE,
"address1": "Some street",
"clli": str(uuid.uuid4())[:6].upper(),
"rencode": str(uuid.uuid4())[:6].upper(),
"rencode": "",
"npanxx": "000-111",
"latitude": None,
"longitude": None,
@@ -333,6 +334,7 @@ class TestJSON(unittest.TestCase):
"policy_locations": "Required - International",
"policy_ratio": True,
"policy_contracts": "Required",
"allow_ixp_update": True,
}
data.update(**kwargs)
return data
@@ -365,6 +367,7 @@ class TestJSON(unittest.TestCase):
"descr": NOTE,
"mtu": 12345,
"dot1q_support": False,
"ixf_ixp_member_list_url_visible": "Private",
"rs_asn": 12345,
"arp_sponge": None,
}
@@ -381,7 +384,7 @@ class TestJSON(unittest.TestCase):
"ixlan_id": SHARED["ixlan_r_ok"].id,
"protocol": "IPv4",
"prefix": "10.%d.10.0/23" % (self.PREFIX_COUNT + 1),
"in_dfz": False
"in_dfz": False,
}
if "prefix" not in kwargs:
self.PREFIX_COUNT += 1
@@ -506,7 +509,9 @@ class TestJSON(unittest.TestCase):
##########################################################################
def assert_delete(self, db, typ, test_success=None, test_failure=None):
def assert_delete(
self, db, typ, test_success=None, test_failure=None, test_protected=None
):
if test_success:
db.rm(typ, test_success)
with pytest.raises(NotFoundException):
@@ -520,6 +525,13 @@ class TestJSON(unittest.TestCase):
except PermissionDeniedException:
pass
if test_protected:
with pytest.raises(PermissionDeniedException):
db.rm(typ, test_protected)
assert DeskProTicket.objects.filter(
subject__icontains=f"{typ}-{test_protected}"
).exists()
##########################################################################
def assert_create(
@@ -553,9 +565,12 @@ class TestJSON(unittest.TestCase):
with pytest.raises(InvalidRequestException) as excinfo:
r = db.create(typ, data_invalid, return_response=True)
# FIXME
# The following lines will not be called since
# the InvalidRequestException is raised
# in the previous line
for k, v in list(test_failures["invalid"].items()):
self.assertIn(k, list(r.keys()))
assert "400 Bad Request" in str(excinfo.value)
# we test fail because of parent entity status
@@ -921,6 +936,17 @@ class TestJSON(unittest.TestCase):
##########################################################################
def test_user_001_GET_ixlan_ixf_ixp_member_list_url(self):
for ixlan in self.db_user.all(
"ixlan", ixf_ixp_member_list_url__startswith="http"
):
if ixlan["ixf_ixp_member_list_url_visible"] in ["Public", "Users"]:
assert ixlan["ixf_ixp_member_list_url"] == "http://localhost"
else:
assert "ixf_ixp_member_list_url" not in ixlan
##########################################################################
def test_user_001_GET_ixpfx(self):
self.assert_get_handleref(self.db_user, "ixpfx", SHARED["ixpfx_r_ok"].id)
@@ -965,6 +991,19 @@ class TestJSON(unittest.TestCase):
self.db_org_member, "poc", SHARED["poc_r_ok_private"].id
)
#########################################################################
def test_org_member_001_GET_ixlan_ixf_ixp_member_list_url(self):
for ixlan in self.db_org_member.all(
"ixlan", ixf_ixp_member_list_url__startswith="http"
):
if ixlan["ixf_ixp_member_list_url_visible"] in ["Public", "Users"]:
assert ixlan["ixf_ixp_member_list_url"] == "http://localhost"
else:
if ixlan["id"] == SHARED["ixlan_r3_ok"].id:
assert ixlan["ixf_ixp_member_list_url"] == "http://localhost"
else:
assert "ixf_ixp_member_list_url" not in ixlan
##########################################################################
# TESTS WITH USER THAT IS ORGANIZATION ADMINISTRATOR
##########################################################################
@@ -990,6 +1029,20 @@ class TestJSON(unittest.TestCase):
self.db_org_admin, "poc", SHARED["poc_r_ok_private"].id
)
#########################################################################
def test_org_admin_001_GET_ixlan_ixf_ixp_member_list_url(self):
for ixlan in self.db_org_admin.all(
"ixlan", ixf_ixp_member_list_url__startswith="http"
):
if ixlan["ixf_ixp_member_list_url_visible"] in ["Public", "Users"]:
assert ixlan["ixf_ixp_member_list_url"] == "http://localhost"
else:
if ixlan["id"] == SHARED["ixlan_rw3_ok"].id:
assert ixlan["ixf_ixp_member_list_url"] == "http://localhost"
else:
assert "ixf_ixp_member_list_url" not in ixlan
##########################################################################
def test_org_admin_002_POST_PUT_DELETE_ix(self):
@@ -1039,6 +1092,7 @@ class TestJSON(unittest.TestCase):
test_failures={
"invalid": {"name": ""},
"perms": {"id": SHARED["ix_r_ok"].id},
"readonly": {"ixf_net_count": 50, "ixf_last_import": "not even valid"},
},
)
@@ -1079,6 +1133,11 @@ class TestJSON(unittest.TestCase):
data = self.make_data_ix(prefix=self.get_prefix6())
self.assert_create(self.db_org_admin, "ix", data, ignore=["prefix"])
# check protected ix validation
self.assert_delete(
self.db_org_admin, "ix", test_protected=SHARED["ix_rw_ok"].id,
)
##########################################################################
def test_org_admin_002_POST_PUT_DELETE_fac(self):
@@ -1089,7 +1148,7 @@ class TestJSON(unittest.TestCase):
"fac",
data,
test_failures={
"invalid": {"name": ""},
"invalid": {"name": "",},
"perms": {
# need to set name again so it doesnt fail unique validation
"name": self.make_name("Test"),
@@ -1116,6 +1175,9 @@ class TestJSON(unittest.TestCase):
"readonly": {
"latitude": 1, # this should not take as it is read only
"longitude": 1, # this should not take as it is read only
"rencode": str(uuid.uuid4())[
:6
].upper(), # this should not take as it is read only
},
},
)
@@ -1127,6 +1189,23 @@ class TestJSON(unittest.TestCase):
test_failure=SHARED["fac_r_ok"].id,
)
# check protected ix validation
self.assert_delete(
self.db_org_admin, "fac", test_protected=SHARED["fac_rw_ok"].id,
)
# Create new data with a non-null rencode
data_new = self.make_data_fac()
obsolete_rencode = str(uuid.uuid4())[:6].upper()
data_new["rencode"] = obsolete_rencode
# Data should be successfully created
r_data_new = self.assert_get_single(
self.db_org_admin.create("fac", data_new, return_response=True).get("data")
)
# But rencode should be null
assert r_data_new["rencode"] == ""
##########################################################################
@@ -1167,6 +1246,15 @@ class TestJSON(unittest.TestCase):
},
)
# Test ASN cannot update
self.assert_update(
self.db_org_admin,
"net",
SHARED["net_id"],
data,
test_failures={"invalid": {"asn": data["asn"] + 1},},
)
self.assert_delete(
self.db_org_admin,
"net",
@@ -1187,24 +1275,28 @@ class TestJSON(unittest.TestCase):
##########################################################################
def test_org_admin_002_POST_net_looking_glass_url(self):
for scheme in ["http","https","ssh","telnet"]:
for scheme in ["http", "https", "ssh", "telnet"]:
r_data = self.assert_create(
self.db_org_admin,
"net",
self.make_data_net(asn=9000900, looking_glass="{}://foo.bar".format(scheme)),
test_failures={"invalid": {"looking_glass": "foo://www.bar.com"}}
self.make_data_net(
asn=9000900, looking_glass="{}://foo.bar".format(scheme)
),
test_failures={"invalid": {"looking_glass": "foo://www.bar.com"}},
)
Network.objects.get(id=r_data["id"]).delete(hard=True)
##########################################################################
def test_org_admin_002_POST_net_route_server_url(self):
for scheme in ["http","https","ssh","telnet"]:
for scheme in ["http", "https", "ssh", "telnet"]:
r_data = self.assert_create(
self.db_org_admin,
"net",
self.make_data_net(asn=9000900, route_server="{}://foo.bar".format(scheme)),
test_failures={"invalid": {"route_server": "foo://www.bar.com"}}
self.make_data_net(
asn=9000900, route_server="{}://foo.bar".format(scheme)
),
test_failures={"invalid": {"route_server": "foo://www.bar.com"}},
)
Network.objects.get(id=r_data["id"]).delete(hard=True)
@@ -1271,32 +1363,6 @@ class TestJSON(unittest.TestCase):
##########################################################################
def test_org_admin_002_PUT_net_write_only_fields(self):
"""
with this we check that certain fields that are allowed to be
set via the api, but sre not supposed to be rendered in the
api data, work correctly
"""
def test_write_only_fields_missing(orig, updated):
assert ("allow_ixp_update" in updated) == False
net = SHARED["net_rw_ok"]
self.assertEqual(net.allow_ixp_update, False)
self.assert_update(
self.db_org_admin,
"net",
net.id,
{"allow_ixp_update": True},
test_success=[test_write_only_fields_missing],
)
net.refresh_from_db()
self.assertEqual(net.allow_ixp_update, True)
##########################################################################
def test_org_admin_002_POST_PUT_DELETE_netfac(self):
data = {
@@ -1478,7 +1544,6 @@ class TestJSON(unittest.TestCase):
},
)
self.assert_delete(
self.db_org_admin,
"ixpfx",
@@ -1491,21 +1556,15 @@ class TestJSON(unittest.TestCase):
# re-delete
self.assert_delete(self.db_org_admin, "ixpfx", test_success=SHARED["ixpfx_id"])
# re-creating a deleted ixpfx that we dont have write permissions do
# should fail
# re-creating a deleted ixpfx that is under another exchange
# that we dont have write perms too
pfx = IXLanPrefix.objects.create(
ixlan=SHARED["ixlan_r_ok"], prefix="205.127.237.0/24", protocol="IPv4"
)
pfx.delete()
data.update(prefix="205.127.237.0/24")
r_data = self.assert_create(
self.db_org_admin,
"ixpfx",
data,
test_failures={"invalid": {}},
test_success=False,
)
r_data = self.assert_create(self.db_org_admin, "ixpfx", data,)
# make sure protocols are validated
r_data = self.assert_create(
@@ -1518,6 +1577,21 @@ class TestJSON(unittest.TestCase):
test_success=False,
)
# test protected ixpfx cant be deleted
prefix = IXLanPrefix.objects.get(id=SHARED["ixpfx_id"])
NetworkIXLan.objects.create(
network=SHARED["net_rw_ok"],
asn=SHARED["net_rw_ok"].asn,
ixlan=SHARED["ixlan_rw_ok"],
ipaddr4=prefix.prefix[0],
status="ok",
speed=1000,
)
self.assert_delete(
self.db_org_admin, "ixpfx", test_protected=SHARED["ixpfx_id"]
)
##########################################################################
def test_org_admin_002_POST_PUT_DELETE_netixlan(self):
@@ -1655,7 +1729,6 @@ class TestJSON(unittest.TestCase):
"org",
# can delete the org we just made
test_success=org.id,
# cant delete the org we dont have write perms to
test_failure=SHARED["org_r_ok"].id,
)
@@ -1728,6 +1801,17 @@ class TestJSON(unittest.TestCase):
##########################################################################
def test_guest_001_GET_ixlan_ixf_ixp_member_list_url(self):
for ixlan in self.db_guest.all(
"ixlan", ixf_ixp_member_list_url__startswith="http"
):
if ixlan["ixf_ixp_member_list_url_visible"] == "Public":
assert ixlan["ixf_ixp_member_list_url"] == "http://localhost"
else:
assert "ixf_ixp_member_list_url" not in ixlan
##########################################################################
def test_guest_001_GET_ixpfx(self):
self.assert_get_handleref(self.db_guest, "ixpfx", SHARED["ixpfx_r_ok"].id)
@@ -2390,7 +2474,7 @@ class TestJSON(unittest.TestCase):
# set one netixlan to not operational
netixlan = NetworkIXLan.objects.first()
netixlan.operational=False
netixlan.operational = False
netixlan.save()
# assert that it is now returned in the operational=False
@@ -2400,7 +2484,6 @@ class TestJSON(unittest.TestCase):
assert len(data) == 1
assert data[0]["id"] == netixlan.id
##########################################################################
def test_guest_005_list_filter_netixlan_related_name(self):
@@ -2432,7 +2515,7 @@ class TestJSON(unittest.TestCase):
def test_guest_005_list_filter_netfac_related_country(self):
data = self.db_guest.all(
"netfac", country=u"{}".format(SHARED["fac_rw_ok"].country)
"netfac", country="{}".format(SHARED["fac_rw_ok"].country)
)
self.assertEqual(len(data), 2)
self.assert_data_integrity(data[0], "netfac")
@@ -2918,6 +3001,50 @@ class TestJSON(unittest.TestCase):
# MISC TESTS
##########################################################################
def _test_GET_ixf_ixp_member_list_url(self, db, tests=[], suffix="r"):
ixlan = SHARED[f"ixlan_{suffix}_ok"]
ixlan.ixf_ixp_member_list_url = "http://localhost"
ixlan.save()
for visible, expected in tests:
ixlan.ixf_ixp_member_list_url_visible = visible
ixlan.full_clean()
ixlan.save()
data = db.get("ixlan", id=ixlan.id)[0]
assert data["ixf_ixp_member_list_url_visible"] == visible
if expected:
assert data["ixf_ixp_member_list_url"] == ixlan.ixf_ixp_member_list_url
else:
assert "ixf_ixp_member_list_url" not in data
def test_z_misc_GET_ixf_ixp_member_list_url(self):
"""
Test the visibility of ixlan.ixf_ixp_member_list_url for
Guest, User, Org member and org admin
"""
self._test_GET_ixf_ixp_member_list_url(
self.db_user, [("Private", False), ("Users", True), ("Public", True)]
)
self._test_GET_ixf_ixp_member_list_url(
self.db_guest, [("Private", False), ("Users", False), ("Public", True)]
)
self._test_GET_ixf_ixp_member_list_url(
self.db_org_member, [("Private", True), ("Users", True), ("Public", True)]
)
self._test_GET_ixf_ixp_member_list_url(
self.db_org_admin,
[("Private", True), ("Users", True), ("Public", True)],
suffix="rw",
)
def test_z_misc_POST_ix_fac_missing_phone_fields(self):
"""
@@ -2945,7 +3072,6 @@ class TestJSON(unittest.TestCase):
del data["policy_phone"]
r = db.create("ix", data, return_response=True).get("data")
def test_z_misc_002_dupe_netixlan_ip(self):
# test that addint duplicate netixlan ips is impossible
@@ -3489,11 +3615,25 @@ class Command(BaseCommand):
org_id=SHARED["org_rw_ok"].id,
)
visibility = {
"rw": "Public",
"rw2": "Users",
"rw3": "Private",
"r": "Public",
"r2": "Users",
"r3": "Private",
}
for status in ["ok", "pending"]:
for prefix in ["r", "rw"]:
SHARED["ixlan_{}_{}".format(prefix, status)] = SHARED[
for prefix in ["r", "r2", "r3", "rw", "rw2", "rw3"]:
ixlan = SHARED["ixlan_{}_{}".format(prefix, status)] = SHARED[
"ix_{}_{}".format(prefix, status)
].ixlan
if prefix in visibility:
visible = visibility[prefix]
ixlan.ixf_ixp_member_list_url_visible = visible
ixlan.ixf_ixp_member_list_url = "http://localhost"
ixlan.save()
for status in ["ok", "pending"]:
for prefix in ["r", "rw"]:

5
peeringdb_server/management/commands/pdb_delete_pocs.py
View File

@@ -8,6 +8,7 @@ from reversion.models import Version
from peeringdb_server.models import NetworkContact
class Command(BaseCommand):
help = "Hard deletes old soft-deleted network contact instances"
@@ -27,7 +28,6 @@ class Command(BaseCommand):
self.commit = options.get("commit")
self._hard_delete_old_pocs()
def _hard_delete_old_pocs(self):
qset = NetworkContact.objects.filter(status="deleted")
@@ -42,7 +42,6 @@ class Command(BaseCommand):
counter = 1
for poc in qset:
self.log(f"Deleting poc={poc.id} ... {counter} / {count}")
@@ -58,5 +57,3 @@ class Command(BaseCommand):
# remove poc
poc.delete(hard=True)

29
peeringdb_server/management/commands/pdb_ixf_ixp_member_import.py
View File

@@ -8,6 +8,7 @@ from peeringdb_server.models import (
IXLan,
NetworkIXLan,
Network,
IXFMemberData,
)
from peeringdb_server import ixf
@@ -36,6 +37,11 @@ class Command(BaseCommand):
action="store_true",
help="Just update IX-F cache, do NOT perform any import logic",
)
parser.add_argument(
"--delete-all-ixfmemberdata",
action="store_true",
help="This removes all IXFMemberData objects",
)
def log(self, msg, debug=False):
if self.preview:
@@ -54,6 +60,10 @@ class Command(BaseCommand):
self.cache = options.get("cache", False)
self.skip_import = options.get("skip_import", False)
if options.get("delete_all_ixfmemberdata"):
self.log("Deleting IXFMemberData Instances ...")
IXFMemberData.objects.all().delete()
if self.preview and self.commit:
self.commit = False
@@ -86,22 +96,23 @@ class Command(BaseCommand):
importer = ixf.Importer()
importer.skip_import = self.skip_import
importer.cache_only = self.cache
self.log("Updating {}".format(ixlan))
self.log(f"Processing {ixlan.ix.name} ({ixlan.id})")
with transaction.atomic():
success, netixlans, netixlans_deleted, log = importer.update(
ixlan, save=self.commit, asn=asn
)
self.log(json.dumps(log), debug=True)
success = importer.update(ixlan, save=self.commit, asn=asn)
self.log(json.dumps(importer.log), debug=True)
self.log(
"Done: {} updated: {} deleted: {}".format(
success, len(netixlans), len(netixlans_deleted)
"Success: {}, added: {}, updated: {}, deleted: {}".format(
success,
len(importer.actions_taken["add"]),
len(importer.actions_taken["modify"]),
len(importer.actions_taken["delete"]),
)
)
total_log["data"].extend(log["data"])
total_log["data"].extend(importer.log["data"])
total_log["errors"].extend(
[
"{}({}): {}".format(ixlan.ix.name, ixlan.id, err)
for err in log["errors"]
for err in importer.log["errors"]
]
)

2
peeringdb_server/management/commands/pdb_stats.py
View File

@@ -106,7 +106,7 @@ class Command(BaseCommand):
`dict` with `stats` and `dt` keys
"""
dt = datetime.datetime.now().replace(tzinfo=UTC())
dt = datetime.datetime.now(datetime.timezone.utc)
stats = self.stats(dt)