.. _hdag: Hyper-Distributed Application Graph =========================================== The HDAG is the top-most artifact used by the SMO in order to deploy an HDA as a combination of vertical apps, VOs and cVOs (as shown in the figure). It provides information about deployment which combines custom intent-orchestration mechanism with the configuration of the services for the specific deployment. .. figure:: ../images/highlevel-hdag-with-vos.png :align: center **HDAG overview** The HDAG descriptor provides the key elements which contain the required information for the NEPHELE Platform to orchestrate an HDA, including intent, monitoring, artifact references and instantiation-time customizations. Technologies -------------- The HDAG is a simple descriptor which only requires the following background information: * `YAML Syntax `_ * `JSONSChema Syntax `_ (You can visualize it `here `_ ) Artifact Structure ------------------ Composed of a single artifact inside a parent folder. .. figure:: ../images/hdag.png :align: center **HDAG artifact** Data Models ------------ The descriptor contains a service chain with all the required information for the Nephele SMO Platform to orchestrate and manage it. e.g., .. code-block:: yaml hdaGraph: imVersion: 0.4.0 id: brussels-demo # SMO needs to create a grounded UUID for this version: "1.0.0" designer: NTUA hdaGraphIntent: security: # Either colocation either VO-related security either app-graph deployed using the same Blueprint enabled: False highAvailability: # Application graph is not considered for reconfigurations. Placement is final enabled: False highPerformance: # Defined end-to-end for all services. If selected, overwrites the per service latency qos intent enabled: False energyEfficiency: # Cost reduction of cloud-deployed services enabled: False description: >- Brussels demo HDAG showcasing a WOT VO services: - id: vo deployment: trigger: auto: dependencies: [] intent: network: deviceProximity: # Only relevant for the VO enabled: False # If true, enable TSN latencies: - connectionPoint: "" # Relevant for the next service in the application graph qos: "best-effort" # "ultralow" - under 10ms # "low" - 1hop maximum # "best-effort" - default value compute: cpu: "small" # "light" - 0.5 vCPU # "small" - 1 vCPU # "medium" - 4 vCPU # "large" - 8 vCPU ram: "small" # "light" - 500MB RAM # "small" - 1 GB RAM # "medium" - 2 GB RAM # "large" - 8 GB RAM storage: "small" # "small" - 10GB # "medium" - 20GB # "large" - 40GB gpu: enabled: True coLocation: [] # - id: 2 # groundedGraphId: UUID connectionPoints: [noise-reduction, image-detection] metrics: # list of metrics to be scraped by centralized monitoring system. - id: vo-metric # used for events metricName: MetricName #name of metric exposed to prometheus selector: # the target service (label). Cannot be to a pod directly, has to point to the service app: service_node_name endpoint: # why would we need more than one??. a second metric will have an additional entry in this list port: metrics # name of exposed port interval: 1s # scraping period of prometheus artifact: # if tag is not added, newest tag will be selected by the docker, helm and hdarctl CLIs. # removing oci:// because should not customize the descriptor for implementations # The OCI thing is only for helm, SMO/FRM should add it based on the target # e.g., tag has a different way of adding it in helm (":1.0.0" --> "--version 1.0.0"). which would also need a custom entry in the descriptor. Better to have translation of descriptor ociImage: "nephele-hdar.netmode.ece.ntua.gr/brussels-demo/custom-vo:1.0.0" ociConfig: type: VO implementer: WOT ociRun: name: HELM version: v3 valuesOverwrite: voDescriptorOverwrite: {} chartOverwrite: {} - id: noise-reduction deployment: trigger: event: condition: ANY events: - id: main-event source: - serviceId: vo metricId: vo-metric condition: # This needs to be mapped to a prometheus query by the monitoring component promQuery: absent(up{prometheus="kc1/kube-prometheus-operator"}) gracePeriod: 2m description: "High vo-metric" intent: network: deviceProximity: # Only relevant for the VO enabled: False # If true, enable TSN latencies: - connectionPoint: "" # Relevant for the next service in the application graph qos: "best-effort" compute: cpu: "small" coLocation: [] # - id: 2 # groundedGraphId: UUID connectionPoints: [vo] artifact: ociImage: "nephele-hdar.netmode.ece.ntua.gr/brussels-demo/noise-reduction" ociConfig: type: application # Helm only allows 'library' and 'application' implementer: HELM ociRun: name: HELM version: v3 valuesOverwrite: {} - id: image-detection deployment: trigger: auto: dependencies: - serviceId: vo intent: network: deviceProximity: # Only relevant for the VO enabled: False # If true, enable TSN latencies: - connectionPoint: "" # Relevant for the next service in the application graph qos: "best-effort" compute: cpu: "small" coLocation: [] # - id: 2 # groundedGraphId: UUID artifact: ociImage: "nephele-hdar.netmode.ece.ntua.gr/brussels-demo/image-detection" ociConfig: type: application implementer: HELM ociRun: name: HELM version: v3 valuesOverwrite: {} The HDAG descriptor is defined using a JSONSChema to ensure that the Nephele Platform works as expected. .. code-block:: json { "$schema": "http://json-schema.org/draft-07/schema#", "type": "object", "properties": { "hdaGraph": { "type": "object", "properties": { "imVersion": { "type": "string", "enum": [ "0.4.0" ] }, "id": { "type": "string" }, "version": { "type": "string", "pattern": "^(0|[1-9]\\d*)\\.(0|[1-9]\\d*)\\.(0|[1-9]\\d*)(?:-((?:0|[1-9]\\d*|\\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\\.(?:0|[1-9]\\d*|\\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\\+([0-9a-zA-Z-]+(?:\\.[0-9a-zA-Z-]+)*))?$" }, "designer": { "type": "string" }, "description": { "type": "string" }, "hdaGraphIntent":{ "type": "object", "properties": { "security": { "type": "object", "properties": { "enabled": { "type": "boolean", "description": "Either colocation either VO-related security either app-graph deployed using the same Blueprint" } }, "required": ["enabled"], "description": "Security settings related to colocation, VO-related security, or app-graph deployment" }, "highAvailability": { "type": "object", "properties": { "enabled": { "type": "boolean", "description": "Application graph is not considered for reconfigurations. Placement is final" } }, "required": ["enabled"], "description": "High availability settings for the application graph" }, "highPerformance": { "type": "object", "properties": { "enabled": { "type": "boolean", "description": "Defined end-to-end for all services. If selected, overwrites the per service latency qos intent" } }, "required": ["enabled"], "description": "High performance settings for all services, potentially overwriting service-specific latency QoS intent" }, "energyEfficiency": { "type": "object", "properties": { "enabled": { "type": "boolean", "description": "Cost reduction of cloud-deployed services" } }, "required": ["enabled"], "description": "Energy efficiency settings for reducing the cost of cloud-deployed services" } }, "required": ["security", "highAvailability", "highPerformance", "energyEfficiency"], "description": "Configuration object for various service settings related to security, availability, performance, and energy efficiency" }, "services": { "type": "array", "minItems": 1, "items": { "type": "object", "properties": { "id": { "type": "string" }, "deployment": { "type": "object", "properties": { "trigger": { "type": "object", "oneOf": [ { "properties": { "auto": { "type": "object", "properties": { "dependencies": { "type": "array", "minItems": 0, "items": { "type": "object", "properties": { "serviceId": { "type": "string" } }, "additionalProperties": false, "required": [ "serviceId" ] } } }, "additionalProperties": false } }, "additionalProperties": false }, { "properties": { "event": { "type": "object", "properties": { "condition": { "type": "string" }, "events": { "type": "array", "minItems": 1, "items": { "type": "object", "properties": { "id": { "type": "string" }, "source": { "type": "array", "minItems": 1, "items": { "type": "object", "properties": { "serviceId": { "type": "string" }, "metricId": { "type": "string" } }, "additionalProperties": false, "required": [ "serviceId", "metricId" ] } }, "condition": { "type": "object", "properties": { "promQuery": { "type": "string" }, "description": { "type": "string" }, "gracePeriod": { "type": "string" } }, "required": [ "gracePeriod", "promQuery", "description" ] } }, "additionalProperties": false, "required": [ "id", "source", "condition" ] } } }, "required": [ "condition", "events" ], "additionalProperties": false } }, "required": [ "event" ], "additionalProperties": false } ] }, "intent": { "type": "object", "properties": { "network": { "type": "object", "properties": { "deviceProximity": { "type": "object", "properties": { "enabled": { "type": "boolean", "description": "If true, enable TSN. Only relevant for the VO" } }, "required": ["enabled"], "description": "Device proximity settings for VO" }, "latencies": { "type": "array", "items": { "type": "object", "properties": { "connectionPoint": { "type": "string", "description": "Relevant for the next service in the application graph" }, "qos": { "type": "string", "enum": ["ultralow", "low", "best-effort"], "description": "\"ultralow\" - under 10ms, \"low\" - 1hop maximum, \"best-effort\" - default value" } }, "required": ["connectionPoint", "qos"], "description": "Latency settings for the connection point in the application graph" }, "description": "Latency settings related to connection and QoS for the application graph" } }, "required": ["deviceProximity", "latencies"], "description": "Network configurations for services, including device proximity and latency settings" }, "compute": { "type": "object", "properties": { "cpu": { "type": "string", "enum": ["light", "small", "medium", "large"], "description": "\"light\" - 0.5 vCPU, \"small\" - 1 vCPU, \"medium\" - 4 vCPU, \"large\" - 8 vCPU" }, "ram": { "type": "string", "enum": ["light", "small", "medium", "large"], "description": "\"light\" - 500MB RAM, \"small\" - 1 GB RAM, \"medium\" - 2 GB RAM, \"large\" - 8 GB RAM" }, "storage": { "type": "string", "enum": ["small", "medium", "large"], "description": "\"small\" - 10GB, \"medium\" - 20GB, \"large\" - 40GB" }, "gpu": { "type": "object", "properties": { "enabled": { "type": "boolean", "description": "If true, GPU is enabled" } }, "required": ["enabled"], "description": "GPU settings for compute resources" } }, "required": ["cpu", "ram", "storage", "gpu"], "description": "Compute resource settings, including CPU, RAM, storage, and GPU configurations" }, "connectionPoints": { "type": "array", "minItems": 0, "items": { "type": "string" } }, "coLocation": { "type": "array", "minItems": 0, "items": { "type": "object" }, "properties": { "id": { "type": "string" }, "groundedGraphId": { "type": "string" } }, "required": [ "id" ], "additionalProperties": false }, "metrics": { "type": "array", "items": { "type": "object", "properties": { "id": { "type": "string", "description": "Identifier used for events" }, "metricName": { "type": "string", "description": "Name of metric exposed to Prometheus" }, "selector": { "type": "object", "description": "The target service (label). Cannot be to a pod directly, has to point to the service", "additionalProperties": { "type": "string" } }, "endpoint": { "type": "object", "description": "Information about the endpoint", "properties": { "port": { "type": "string", "description": "Name of exposed port" }, "interval": { "type": "string", "description": "Scraping period of Prometheus" } }, "required": [ "port", "interval" ], "additionalProperties": false } }, "required": [ "id", "metricName", "selector", "endpoint" ], "additionalProperties": false } } }, "required": [ "compute", "network", "coLocation", "connectionPoints" ], "additionalProperties": false } }, "required": [ "trigger", "intent" ], "additionalProperties": true }, "artifact": { "type": "object", "properties": { "ociImage": { "type": "string" }, "ociConfig": { "type": "object", "properties": { "type": { "type": "string" }, "implementer": { "type": "string" } }, "required": [ "type", "implementer" ], "additionalProperties": false }, "ociRun": { "type": "object", "properties": { "name": { "type": "string" }, "version": { "type": "string" } }, "required": [ "name", "version" ], "additionalProperties": false } }, "required": [ "ociImage", "ociConfig", "ociRun" ], "additionalProperties": true } }, "required": [ "id", "deployment", "artifact" ] } } }, "additionalProperties": false, "required": [ "imVersion", "designer", "version", "hdaGraphIntent", "description", "services" ] } }, "additionalProperties": false, "required": [ "hdaGraph" ] } OCI manifests --------------- Image manifest ################# It follows the standard container manifest structure. E.g. .. code-block:: json { "schemaVersion": 2, "mediaType": "application/vnd.oci.image.manifest.v1+json", "config": { "mediaType": "application/vnd.nephele.hdag.config.v1+json", "digest": "sha256:8bcbbfb98ab93d80fb25343aabf46efe595fb05b21ad97d1e394ee5e08b30ae0", "size": 421 }, "layers": [ { "mediaType": "application/tar+gzip", "digest": "sha256:06698addaa5d783bacfb18c2ecddf09bb3b7c2c846fbe09a4db003f4ecaad352", "size": 1117, "annotations": { "org.opencontainers.image.title": "myfirsthdag-0.1.0.tar.gz" } } ], "annotations": { "org.opencontainers.image.authors": "NEPHELE", "org.opencontainers.image.created": "2024-09-23T15:01:20Z", "org.opencontainers.image.description": "This is my first artifact", "org.opencontainers.image.title": "myfirsthdag", "org.opencontainers.image.version": "0.1.0" } } Config manifest ################# The Config OCI manifest is what Nephele uses to characterize the HDAG without the need of extracting all the contents from the artifact. HDAGs have the **application/vnd.nephele.hdag.config.v1+json** mediaType. E.g., .. code-block:: json { "schemaVersion": "2", "configMediaType": "application/vnd.nephele.hdag.config.v1+json", "artifactType": "HDAG", "dataModelSpec": "0.4.0", "descriptor": { "name": "myfirsthdag/descriptor.yaml", "id": "myfirsthdag", "version": "0.1.0" }, "dependencies": [ { "artifactType": "VO", "ref": "https://nephele-hdar.netmode.ece.ntua.gr/PLACEHOLDER", "location": "external", "mode": "hard" } ] } This json follows a standard structure defined by the following jsonschema .. code-block:: json { "$schema": "http://json-schema.org/draft-07/schema#", "title": "HDAG Configuration Schema", "type": "object", "properties": { "schemaVersion": { "type": "string", "enum": ["0.4.0"], "description": "Schema version" }, "configMediaType": { "type": "string", "enum": ["application/vnd.nephele.hdag.config.v1+json"], "description": "Media type for the HDAG configuration." }, "artifactType": { "type": "string", "enum": ["HDAG"], "description": "Type of the artifact, must be 'HDAG'." }, "dataModelSpec": { "type": "string", "pattern": "^[0-9]+\\.[0-9]+\\.[0-9]+$", "description": "Data model specification version in semantic version format." }, "descriptor": { "type": "object", "properties": { "name": { "type": "string", "description": "Path to the descriptor file." }, "id": { "type": "string", "description": "ID of the descriptor." }, "version": { "type": "string", "pattern": "^[0-9]+\\.[0-9]+\\.[0-9]+$", "description": "Version of the descriptor in semantic version format." } }, "required": ["name", "id", "version"] }, "dependencies": { "type": "array", "items": { "type": "object", "properties": { "artifactType": { "type": "string", "enum": ["VO"], "description": "Artifact type, must be 'VO'." }, "ref": { "type": "string", "format": "uri", "description": "Reference URL for the dependency." }, "location": { "type": "string", "enum": ["external", "internal"], "description": "Location of the dependency." }, "mode": { "type": "string", "enum": ["hard", "soft"], "description": "Mode of the dependency." } }, "required": ["artifactType", "ref", "location", "mode"] } } }, "required": [ "schemaVersion", "configMediaType", "artifactType", "dataModelSpec", "descriptor", "dependencies" ] } .. _how_to_create_hdag: How to create one ----------------- The creation of an HDAG can be done in two ways: #. The create command of the hdarctl tool offers the options shown in the figure below to obtain a baseline descriptor, which only requires a few additional configurations to be pushed to the HDAR. #. The Dashboard offers a UI which presents a user-friendly and intuitive form to generate the baseline HDAG in an analogous way as what the hdarctl does. Then, the user can configure the remaining details via the UI, allowing the Dashboard to interact with the HDAR and automatically push the configurations. .. figure:: ../images/hdagcreate1.png :align: center **hdarctl create comand**