Closes #14436: Add indexes for all GenericForeignKey fields (#14463)

* Closes #14436: Add PostgreSQL indexes for all GenericForeignKeys

* Add note about GFK indexes to developer docs

docs/development/extending-models.md

@@ -2,12 +2,25 @@
 
 Below is a list of tasks to consider when adding a new field to a core model.
 
-## 1. Generate and run database migrations
+## 1. Add the field to the model class
+
+Add the field to the model, taking care to address any of the following conditions.
+
+* When adding a GenericForeignKey field, also add an index under `Meta` for its two concrete fields. For example:
+
+    ```python
+    class Meta:
+        indexes = (
+            models.Index(fields=('object_type', 'object_id')),
+        )
+    ```
+
+## 2. Generate and run database migrations
 
 [Django migrations](https://docs.djangoproject.com/en/stable/topics/migrations/) are used to express changes to the database schema. In most cases, Django can generate these automatically, however very complex changes may require manual intervention. Always remember to specify a short but descriptive name when generating a new migration.
 
 ```
-./manage.py makemigrations <app> -n <name>
+./manage.py makemigrations <app> -n <name> --no-header
 ./manage.py migrate
 ```
 
@@ -16,7 +29,7 @@ Where possible, try to merge related changes into a single migration. For exampl
 !!! warning "Do not alter existing migrations"
     Migrations can only be merged within a release. Once a new release has been published, its migrations cannot be altered (other than for the purpose of correcting a bug).
 
-## 2. Add validation logic to `clean()`
+## 3. Add validation logic to `clean()`
 
 If the new field introduces additional validation requirements (beyond what's included with the field itself), implement them in the model's `clean()` method. Remember to call the model's original method using `super()` before or after your custom validation as appropriate:
 
@@ -31,15 +44,15 @@ class Foo(models.Model):
             raise ValidationError()
 ```
 
-## 3. Update relevant querysets
+## 4. Update relevant querysets
 
 If you're adding a relational field (e.g. `ForeignKey`) and intend to include the data when retrieving a list of objects, be sure to include the field using `prefetch_related()` as appropriate. This will optimize the view and avoid extraneous database queries.
 
-## 4. Update API serializer
+## 5. Update API serializer
 
 Extend the model's API serializer in `<app>.api.serializers` to include the new field. In most cases, it will not be necessary to also extend the nested serializer, which produces a minimal representation of the model.
 
-## 5. Add fields to forms
+## 6. Add fields to forms
 
 Extend any forms to include the new field(s) as appropriate. These are found under the `forms/` directory within each app. Common forms include:
 
@@ -48,23 +61,23 @@ Extend any forms to include the new field(s) as appropriate. These are found und
 * **CSV import** - The form used when bulk importing objects in CSV format
 * **Filter** - Displays the options available for filtering a list of objects (both UI and API)
 
-## 6. Extend object filter set
+## 7. Extend object filter set
 
 If the new field should be filterable, add it to the `FilterSet` for the model. If the field should be searchable, remember to query it in the FilterSet's `search()` method.
 
-## 7. Add column to object table
+## 8. Add column to object table
 
 If the new field will be included in the object list view, add a column to the model's table. For simple fields, adding the field name to `Meta.fields` will be sufficient. More complex fields may require declaring a custom column. Also add the field name to `default_columns` if the column should be present in the table by default.
 
-## 8. Update the SearchIndex
+## 9. Update the SearchIndex
 
 Where applicable, add the new field to the model's SearchIndex for inclusion in global search.
 
-## 9. Update the UI templates
+## 10. Update the UI templates
 
 Edit the object's view template to display the new field. There may also be a custom add/edit form template that needs to be updated.
 
-## 10. Create/extend test cases
+## 11. Create/extend test cases
 
 Create or extend the relevant test cases to verify that the new field and any accompanying validation logic perform as expected. This is especially important for relational fields. NetBox incorporates various test suites, including:
 
@@ -74,8 +87,8 @@ Create or extend the relevant test cases to verify that the new field and any ac
 * Model tests
 * View tests
 
-Be diligent to ensure all of the relevant test suites are adapted or extended as necessary to test any new functionality.
+Be diligent to ensure all the relevant test suites are adapted or extended as necessary to test any new functionality.
 
-## 11. Update the model's documentation
+## 12. Update the model's documentation
 
 Each model has a dedicated page in the documentation, at `models/<app>/<model>.md`. Update this file to include any relevant information about the new field.

netbox/core/migrations/0010_gfk_indexes.py

@@ -0,0 +1,17 @@
+# Generated by Django 4.2.7 on 2023-12-07 16:09
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('core', '0009_configrevision'),
+    ]
+
+    operations = [
+        migrations.AddIndex(
+            model_name='job',
+            index=models.Index(fields=['object_type', 'object_id'], name='core_job_object__c664ac_idx'),
+        ),
+    ]

netbox/core/models/jobs.py

@@ -106,6 +106,9 @@ class Job(models.Model):
 
     class Meta:
         ordering = ['-created']
+        indexes = (
+            models.Index(fields=('object_type', 'object_id')),
+        )
         verbose_name = _('job')
         verbose_name_plural = _('jobs')
 

netbox/dcim/migrations/0184_gfk_indexes.py

@@ -0,0 +1,25 @@
+# Generated by Django 4.2.7 on 2023-12-07 16:09
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('dcim', '0183_protect_child_interfaces'),
+    ]
+
+    operations = [
+        migrations.AddIndex(
+            model_name='cabletermination',
+            index=models.Index(fields=['termination_type', 'termination_id'], name='dcim_cablet_termina_884752_idx'),
+        ),
+        migrations.AddIndex(
+            model_name='inventoryitem',
+            index=models.Index(fields=['component_type', 'component_id'], name='dcim_invent_compone_0560bb_idx'),
+        ),
+        migrations.AddIndex(
+            model_name='inventoryitemtemplate',
+            index=models.Index(fields=['component_type', 'component_id'], name='dcim_invent_compone_77b5f8_idx'),
+        ),
+    ]

netbox/dcim/models/cables.py

@@ -298,6 +298,9 @@ class CableTermination(ChangeLoggedModel):
 
     class Meta:
         ordering = ('cable', 'cable_end', 'pk')
+        indexes = (
+            models.Index(fields=('termination_type', 'termination_id')),
+        )
         constraints = (
             models.UniqueConstraint(
                 fields=('termination_type', 'termination_id'),

netbox/dcim/models/device_component_templates.py

@@ -749,6 +749,9 @@ class InventoryItemTemplate(MPTTModel, ComponentTemplateModel):
 
     class Meta:
         ordering = ('device_type__id', 'parent__id', '_name')
+        indexes = (
+            models.Index(fields=('component_type', 'component_id')),
+        )
         constraints = (
             models.UniqueConstraint(
                 fields=('device_type', 'parent', 'name'),

netbox/dcim/models/device_components.py

@@ -1250,6 +1250,9 @@ class InventoryItem(MPTTModel, ComponentModel, TrackingModelMixin):
 
     class Meta:
         ordering = ('device__id', 'parent__id', '_name')
+        indexes = (
+            models.Index(fields=('component_type', 'component_id')),
+        )
         constraints = (
             models.UniqueConstraint(
                 fields=('device', 'parent', 'name'),

netbox/extras/migrations/0101_eventrule.py

@@ -91,6 +91,10 @@ class Migration(migrations.Migration):
             name='tags',
             field=taggit.managers.TaggableManager(through='extras.TaggedItem', to='extras.Tag'),
         ),
+        migrations.AddIndex(
+            model_name='eventrule',
+            index=models.Index(fields=['action_object_type', 'action_object_id'], name='extras_even_action__d9e2af_idx'),
+        ),
 
         # Replicate Webhook data
         migrations.RunPython(move_webhooks),

netbox/extras/migrations/0103_gfk_indexes.py

@@ -0,0 +1,37 @@
+# Generated by Django 4.2.7 on 2023-12-07 16:09
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('extras', '0102_move_configrevision'),
+    ]
+
+    operations = [
+        migrations.AddIndex(
+            model_name='bookmark',
+            index=models.Index(fields=['object_type', 'object_id'], name='extras_book_object__2df6b4_idx'),
+        ),
+        migrations.AddIndex(
+            model_name='imageattachment',
+            index=models.Index(fields=['content_type', 'object_id'], name='extras_imag_content_94728e_idx'),
+        ),
+        migrations.AddIndex(
+            model_name='journalentry',
+            index=models.Index(fields=['assigned_object_type', 'assigned_object_id'], name='extras_jour_assigne_76510f_idx'),
+        ),
+        migrations.AddIndex(
+            model_name='objectchange',
+            index=models.Index(fields=['changed_object_type', 'changed_object_id'], name='extras_obje_changed_927fe5_idx'),
+        ),
+        migrations.AddIndex(
+            model_name='objectchange',
+            index=models.Index(fields=['related_object_type', 'related_object_id'], name='extras_obje_related_bfcdef_idx'),
+        ),
+        migrations.AddIndex(
+            model_name='stagedchange',
+            index=models.Index(fields=['object_type', 'object_id'], name='extras_stag_object__4734d5_idx'),
+        ),
+    ]

netbox/extras/models/change_logging.py

@@ -94,6 +94,10 @@ class ObjectChange(models.Model):
 
     class Meta:
         ordering = ['-time']
+        indexes = (
+            models.Index(fields=('changed_object_type', 'changed_object_id')),
+            models.Index(fields=('related_object_type', 'related_object_id')),
+        )
         verbose_name = _('object change')
         verbose_name_plural = _('object changes')
 

netbox/extras/models/models.py

@@ -132,6 +132,9 @@ class EventRule(CustomFieldsMixin, ExportTemplatesMixin, TagsMixin, ChangeLogged
 
     class Meta:
         ordering = ('name',)
+        indexes = (
+            models.Index(fields=('action_object_type', 'action_object_id')),
+        )
         verbose_name = _('event rule')
         verbose_name_plural = _('event rules')
 
@@ -631,6 +634,9 @@ class ImageAttachment(ChangeLoggedModel):
 
     class Meta:
         ordering = ('name', 'pk')  # name may be non-unique
+        indexes = (
+            models.Index(fields=('content_type', 'object_id')),
+        )
         verbose_name = _('image attachment')
         verbose_name_plural = _('image attachments')
 
@@ -720,6 +726,9 @@ class JournalEntry(CustomFieldsMixin, CustomLinksMixin, TagsMixin, ExportTemplat
 
     class Meta:
         ordering = ('-created',)
+        indexes = (
+            models.Index(fields=('assigned_object_type', 'assigned_object_id')),
+        )
         verbose_name = _('journal entry')
         verbose_name_plural = _('journal entries')
 
@@ -769,6 +778,9 @@ class Bookmark(models.Model):
 
     class Meta:
         ordering = ('created', 'pk')
+        indexes = (
+            models.Index(fields=('object_type', 'object_id')),
+        )
         constraints = (
             models.UniqueConstraint(
                 fields=('object_type', 'object_id', 'user'),

netbox/extras/models/staging.py

@@ -90,6 +90,9 @@ class StagedChange(ChangeLoggedModel):
 
     class Meta:
         ordering = ('pk',)
+        indexes = (
+            models.Index(fields=('object_type', 'object_id')),
+        )
         verbose_name = _('staged change')
         verbose_name_plural = _('staged changes')
 

netbox/ipam/migrations/0069_gfk_indexes.py

@@ -0,0 +1,25 @@
+# Generated by Django 4.2.7 on 2023-12-07 16:09
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('ipam', '0068_move_l2vpn'),
+    ]
+
+    operations = [
+        migrations.AddIndex(
+            model_name='fhrpgroupassignment',
+            index=models.Index(fields=['interface_type', 'interface_id'], name='ipam_fhrpgr_interfa_2acc3f_idx'),
+        ),
+        migrations.AddIndex(
+            model_name='ipaddress',
+            index=models.Index(fields=['assigned_object_type', 'assigned_object_id'], name='ipam_ipaddr_assigne_890ab8_idx'),
+        ),
+        migrations.AddIndex(
+            model_name='vlangroup',
+            index=models.Index(fields=['scope_type', 'scope_id'], name='ipam_vlangr_scope_t_9da557_idx'),
+        ),
+    ]

netbox/ipam/models/fhrp.py

@@ -101,6 +101,9 @@ class FHRPGroupAssignment(ChangeLoggedModel):
 
     class Meta:
         ordering = ('-priority', 'pk')
+        indexes = (
+            models.Index(fields=('interface_type', 'interface_id')),
+        )
         constraints = (
             models.UniqueConstraint(
                 fields=('interface_type', 'interface_id', 'group'),

netbox/ipam/models/ip.py

@@ -780,9 +780,10 @@ class IPAddress(PrimaryModel):
 
     class Meta:
         ordering = ('address', 'pk')  # address may be non-unique
-        indexes = [
+        indexes = (
             models.Index(Cast(Host('address'), output_field=IPAddressField()), name='ipam_ipaddress_host'),
-        ]
+            models.Index(fields=('assigned_object_type', 'assigned_object_id')),
+        )
         verbose_name = _('IP address')
         verbose_name_plural = _('IP addresses')
 

netbox/ipam/models/vlans.py

@@ -68,6 +68,9 @@ class VLANGroup(OrganizationalModel):
 
     class Meta:
         ordering = ('name', 'pk')  # Name may be non-unique
+        indexes = (
+            models.Index(fields=('scope_type', 'scope_id')),
+        )
         constraints = (
             models.UniqueConstraint(
                 fields=('scope_type', 'scope_id', 'name'),

netbox/tenancy/migrations/0013_gfk_indexes.py

@@ -0,0 +1,17 @@
+# Generated by Django 4.2.7 on 2023-12-07 16:09
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('tenancy', '0012_contactassignment_custom_fields'),
+    ]
+
+    operations = [
+        migrations.AddIndex(
+            model_name='contactassignment',
+            index=models.Index(fields=['content_type', 'object_id'], name='tenancy_con_content_693ff4_idx'),
+        ),
+    ]

netbox/tenancy/models/contacts.py

@@ -141,6 +141,9 @@ class ContactAssignment(CustomFieldsMixin, ExportTemplatesMixin, TagsMixin, Chan
 
     class Meta:
         ordering = ('priority', 'contact')
+        indexes = (
+            models.Index(fields=('content_type', 'object_id')),
+        )
         constraints = (
             models.UniqueConstraint(
                 fields=('content_type', 'object_id', 'contact', 'role'),

netbox/vpn/migrations/0001_initial.py

@@ -219,6 +219,10 @@ class Migration(migrations.Migration):
                 'ordering': ('tunnel', 'role', 'pk'),
             },
         ),
+        migrations.AddIndex(
+            model_name='tunneltermination',
+            index=models.Index(fields=['termination_type', 'termination_id'], name='vpn_tunnelt_termina_c1f04b_idx'),
+        ),
         migrations.AddConstraint(
             model_name='tunneltermination',
             constraint=models.UniqueConstraint(fields=('termination_type', 'termination_id'), name='vpn_tunneltermination_termination', violation_error_message='An object may be terminated to only one tunnel at a time.'),

netbox/vpn/migrations/0002_move_l2vpn.py

@@ -70,4 +70,8 @@ class Migration(migrations.Migration):
                 name='vpn_l2vpntermination_assigned_object'
             ),
         ),
+        migrations.AddIndex(
+            model_name='l2vpntermination',
+            index=models.Index(fields=['assigned_object_type', 'assigned_object_id'], name='vpn_l2vpnte_assigne_9c55f8_idx'),
+        ),
     ]

netbox/vpn/models/l2vpn.py

@@ -104,6 +104,9 @@ class L2VPNTermination(NetBoxModel):
 
     class Meta:
         ordering = ('l2vpn',)
+        indexes = (
+            models.Index(fields=('assigned_object_type', 'assigned_object_id')),
+        )
         constraints = (
             models.UniqueConstraint(
                 fields=('assigned_object_type', 'assigned_object_id'),

netbox/vpn/models/tunnels.py

@@ -143,6 +143,9 @@ class TunnelTermination(CustomFieldsMixin, CustomLinksMixin, TagsMixin, ChangeLo
 
     class Meta:
         ordering = ('tunnel', 'role', 'pk')
+        indexes = (
+            models.Index(fields=('termination_type', 'termination_id')),
+        )
         constraints = (
             models.UniqueConstraint(
                 fields=('termination_type', 'termination_id'),