Python Panels SDK API¶

This page documents the comet_ml.API for Comet Python Panels.

Other Python Panel Resources:

API ¶

The API class is used as a Python interface to the comet-ml.com Python Panels API.

You can use an instance of the API() class to quickly and easily access all of your logged information at comet-ml.com, including metrics, parameters, tags, and assets.

Example calls to get workspace, project, and experiment data:

API.get(): gets all of your personal workspaces
API.get(WORKSPACE): gets all of your projects from WORKSPACE
API.get(WORKSPACE, PROJECT_NAME): get all APIExperiments in WORKSPACE/PROJECT
API.get_experiment(WORKSPACE, PROJECT_NAME, EXPERIMENT_KEY): get an APIExperiment
API.get_experiment("WORKSPACE/PROJECT_NAME/EXPERIMENT_KEY"): get an APIExperiment
API.get_experiments(WORKSPACE): get all APIExperiments in WORKSPACE
API.get_experiments(WORKSPACE, PROJECT_NAME): get all APIExperiments in WORKSPACE/PROJECT
API.get_experiments(WORKSPACE, PROJECT_NAME, PATTERN): get all APIExperiments in WORKSPACE/PROJECT/PATTERN

Examples:

>>> import comet_ml
>>> api = comet_ml.api.API()

## Return all of my workspace names in a list:
>>> api.get()

## Get an APIExperiment:
>>> experiment = api.get("cometpublic/comet-notebooks/example 001")

## Get metrics:
>>> experiment.get_metrics("train_accuracy")

Within Python Panels, the API() class is limited only to read methods. To log new data to Comet, see the Python SDK.

API.get¶

get(self, workspace=None, project_name=None, experiment=None)

Get the following items:

list of workspace names, given no arguments
list of project names, given a workspace name
list of experiment names/keys, given workspace and project names
an experiment, given workspace, project, and experiment name/key

workspace, project_name, and experiment can also be given as a single string, delimited with a slash.

API.get_experiment¶

get_experiment(self, experiment_key)

Get a single APIExperiment by workspace, project, experiment.

API.get_experiment_asset¶

get_experiment_asset(self, experiment_key, asset_id, return_type="raw")

Get an asset, given the asset_id.

Args:

asset_id: the asset ID
return_type: the type of object returned. Default is "binary". Options: "binary" or "text"

Example:

>>> api_experiment.get_asset("298378237283728", return_type="json")
{...}

API.get_experiment_asset_by_name¶

get_experiment_asset_by_name(self, experiment_key, file_name,
    return_type=None)

Get an asset by name.

API.get_experiment_asset_list¶

get_experiment_asset_list(self, experiment_key, asset_type="all")

Get a list of assets associated with the experiment.

Args:

asset_type: Optional String, type of asset to return. Can be "all", "image", "histogram_combined_3d", "video", or "audio".

Returns a list of dictionaries of asset properties, like:

>>> from comet_ml.api import API
>>> api = API()
>>> api.get_experiment_asset_list(key)
[{'fileName': 'My Filename.png',
'fileSize': 21113,
'runContext': None,
'step': None,
'link': 'https://www.comet-ml.com/api/asset/download?experimentKey=KEY&assetId=ASSET_ID',
'createdAt': 1565898755830,
'dir': 'assets',
'canView': False,
'audio': False,
'video': False,
'histogram': False,
'image': True,
'type': 'image',
'metadata': None,
'assetId': ASSET_ID}, ...]

>>> api.get_experiment_asset_list(key, "image")
[{'fileName': 'My Filename.png',
'fileSize': 21113,
'runContext': None,
'step': None,
'link': 'https://www.comet-ml.com/api/asset/download?experimentKey=KEY&assetId=ASSET_ID',
'createdAt': 1565898755830,
'dir': 'assets',
'canView': False,
'audio': False,
'video': False,
'histogram': False,
'image': True,
'type': 'image',
'metadata': None,
'assetId': ASSET_ID}, ...]

API.get_experiment_by_id¶

get_experiment_by_id(self, experiment)

API.get_experiment_by_key¶

get_experiment_by_key(self, experiment_key)

Get an APIExperiment by experiment key.

API.get_experiment_by_name¶

get_experiment_by_name(self, experiment_name)

Get an APIExperiment by experiment key.

API.get_experiment_code¶

get_experiment_code(self, experiment_key)

Get the associated source code for this experiment.

Example:

>>> api.get_experiment_code(key)
'import comet_ml
experiment = comet_ml.Experiment()
experiment.end()'

API.get_experiment_curves¶

get_experiment_curves(self, experiment_key)

Get all curve assets of an experiment as bytes.

Example:

>>> api.get_experiment_curves(key)
[...]

API.get_experiment_graph¶

get_experiment_graph(self, experiment_key)

Alias of APIExperiment.get_model_graph()

Get the associated graph/model description for this experiment.

Example:

>>> api_experiment.get_model_graph()
{"class_name": "Sequential", ...}

API.get_experiment_html¶

get_experiment_html(self, experiment_key)

Get the HTML associated with the experiment.

API.get_experiment_images¶

get_experiment_images(self, experiment_key)

Get all image assets of an experiment as bytes.

Example:

>>> api.get_experiment_images(key)
[...]

API.get_experiment_installed_packages¶

get_experiment_installed_packages(self, experiment_key)

Get the associated installed packages for this experiment.

Example:

>>> api_experiment.get_installed_packages()
['absl-py==0.8.1', 'adal==1.2.2', 'alabaster==0.7.12', ...]

API.get_experiment_metadata¶

get_experiment_metadata(self, experiment_key)

Returns the JSON metadata for an experiment

Returns:

{
  'archived': False,
  'durationMillis': 7,
  'endTimeMillis': 1586174765277,
  'experimentKey': 'EXPERIMENT-KEY',
  'experimentName': None,
  'fileName': None,
  'filePath': None,
  'optimizationId': None,
  'projectId': 'PROJECT-ID',
  'projectName': 'PROJECT-NAME',
  'running': False,
  'startTimeMillis': 1586174757596,
  'throttle': False,
  'workspaceName': 'WORKSPACE-NAME',
}

API.get_experiment_metric¶

get_experiment_metric(self, experiment_key, metric)

Return an experiment metric values.

Args:

experiment_key: (string), key of experiment
metric: (string), name of a metric

Examples:

>>> from comet_ml.api import API
>>> api = API()
>>> api.get_experiment_metric(key, 'val_loss')

API.get_experiment_metrics_summaries¶

get_experiment_metrics_summaries(self, experiment_key)

Return the experiment metrics summary.

Args:

experiment_key: (string), key of experiment

Examples:

>>> from comet_ml.api import API
>>> api = API()
>>> api.get_experiment_metrics_summaries(key)
[{'name': 'val_loss',
'valueMax': '0.24951280827820302',
'valueMin': '0.13101346811652184',
'valueCurrent': '0.13101346811652184',
'timestampMax': 1558962367938,
'timestampMin': 1558962367938,
'timestampCurrent': 1558962376383,
'stepMax': 500,
'stepMin': 1500,
'stepCurrent': 1500},
...]

API.get_experiment_model_asset_list¶

get_experiment_model_asset_list(self, experiment_key, model_name)

Get an experiment model's asset list by model name.

Args:

model_name: str, the name of the model

Returns: a list of asset dictionaries with these fields:

fileName
fileSize
runContext
step
link
createdAt
dir
canView
audio
histogram
image
type
metadata
assetId

Example:

>>> from comet_ml import API
>>> api = API()
>>> api.get_experiment_model_asset_list(key, "Model Name")
[
{
  "assetId": 74374637463476,
  "audio": False,
  "canView": False,
  "createdAt": 7337347634,
  "dir": "trained-models",
  "fileName": "model.h5",
  "fileSize": 254654,
  "histogram": False,
  "image": False,
  "link": "https://link-to-download-asset-file",
  "metadata": None,
  "remote": False,
  "runContext": "train",
  "step": 54,
  "type": "asset",
}
]

API.get_experiment_model_graph¶

get_experiment_model_graph(self, experiment_key)

Get the associated graph/model description for this experiment.

Example:

>>> api.get_experiment_model_graph(key)
{"class_name": "Sequential", ...}

API.get_experiment_models¶

get_experiment_models(self, experiment_key)

Given an experiment id, return a list of model data associated with an experiment.

Args:

experiment_id: the experiment's key

Returns [{'experimentModelId': 'MODEL-ID' 'experimentKey': 'EXPERIMENT-KEY', 'modelName': 'MODEL-NAME'}, ...]

API.get_experiment_name¶

get_experiment_name(self, experiment_key, max_length=None)

Get the experiment name for this experiment key.

API.get_experiment_others_summaries¶

get_experiment_others_summaries(self, experiment_key)

Get the other items logged in summary form.

Examples:

>>> from comet_ml.api import API
>>> api = API()
>>> api.get_experiment_others_summaries(key)
[{'name': 'trainable_params',
  'valueMax': '712723',
  'valueMin': '712723',
  'valueCurrent': '712723',
  'timestampMax': 1558962363411,
  'timestampMin': 1558962363411,
  'timestampCurrent': 1558962363411},
...]

API.get_experiment_output¶

get_experiment_output(self, experiment_key)

Get the associated standard output for this experiment.

Example:

>>> api.get_experiment_output(key)
"Experiment is live"

API.get_experiment_parameters_summaries¶

get_experiment_parameters_summaries(self, experiment_key)

Return the experiment parameters summary.

Examples:

>>> from comet_ml.api import API
>>> api = API()
>>> api.get_experiment_parameters_summaries()
[{'name': 'batch_size',
  'valueMax': '120',
  'valueMin': '120',
  'valueCurrent': '120',
  'timestampMax': 1558962363411,
  'timestampMin': 1558962363411,
  'timestampCurrent': 1558962363411},
...]

API.get_experiment_tags¶

get_experiment_tags(self, experiment_key)

Get the associated tags for this experiment.

Example:

>>> api.get_experiment_tags(key)
["best"]

API.get_experiment_url¶

get_experiment_url(self, experiment_key)

Get the url of the experiment.

Example:

>>> api.get_experiment_url()
"https://www.comet-ml.com/username/34637643746374637463476"

API.get_experiments¶

get_experiments(self, workspace, project_name=None, pattern=None)

Get APIExperiments by workspace, workspace + project, or workspace + project + regular expression pattern.

API.get_metrics_for_chart¶

get_metrics_for_chart(self, experiments, metrics=[], parameters=[],
    independent_metrics=True, fetch_full=False)

Get multiple metrics and parameters from a set of experiments. This method is designed to make custom charting easier.

Args:

experiments: a list of experiments or experiment keys
metrics: an optional list of metric names (e.g., "loss"); OR
parameters: an optional list of parameter names (e.g., "learning-rate")
independent: Bool, get independent results?
full: Bool, fetch the full result?

Note: you should pass in a list of metric names, or a list of

parameter names, or both.

Returns: a dictionary of experiment keys with the following

structure. {EXPERIMENT_KEY: {'params'} will be None if there are no parameters passed in.

>>> from comet_ml.api import API
>>> api = API()
>>> api.get_metrics_for_chart([experiment_key1, experiment_key2, ...],
  ["loss"], ["hidden_layer_size"])
{EXPERIMENT_KEY: {
'experiment_key': EXPERIMENT_KEY,
'steps': STEPS,
'epochs': None,
'metrics': [
  {'metricName': 'loss',
  'values': [VALUE, ...],
  'steps': [STEP, ...],
  'epochs': [EPOCH, ...],
  'timestamps': [TIMESTAMP, ...],
  'durations': [DURATION, ...],
  }],
'params': {'hidden_layer_size': VALUE, ...},
}, ...}

API.get_model_graph¶

get_model_graph(self, experiment_key)

Get the associated graph/model description for this experiment.

Example:

>>> api_experiment.get_model_graph()
{"class_name": "Sequential", ...}

API.get_panel_experiment_keys¶

get_panel_experiment_keys(self)

Get the current panel's experiment keys.

Example:

from comet_ml import API

api = API()
experiment_keys = api.get_panel_experiment_keys()

This will return a list containing all of the current experiment keys. Specifically:

If on a Project View, the experiment keys from the visible experiment table page
If a filter is set, all of the matching experiments
If a Query is set for this Panel, the matching experiments

API.get_panel_experiments¶

get_panel_experiments(self)

Get the current panel's experiments.

Example:

from comet_ml import API

api = API()
experiment = api.get_panel_experiments()

Like the API.get_experiment_keys() method, but returns a list of APIExperiments rather than just their keys.

API.get_panel_metrics_names¶

get_panel_metrics_names(self, include_sys=False)

Get the current panel's experiments' metrics' names.

Args:

include_sys (bool): if True, include the system metrics

Example:

from comet_ml import API, ui

api = API()
metrics_names = api.get_panel_metrics_names()
metric_name = ui.dropdown("Select a metric name:", metrics_names)

As shown, the metrics_names could be useful in creating a dropdown selection for plotting.

Note that, by default, this list does not includes all metrics. Use include_sys=True to include system metrics as well.

API.get_panel_options¶

get_panel_options(self)

Get the current panel's options as a Python dictionary from the Option tab for this Panel instance.

Example:

from comet_ml import API

api = API()
options = api.get_panel_options()

The Panel Options can be edited when in the Options tab of the Panel Code editor, and set when you are adding a Panel instance to a View.

API.get_panel_project_id¶

get_panel_project_id(self)

Get the current panel's ID.

Example:

from comet_ml import API

api = API()
project_id = api.get_panel_project_id()

The project_id can be useful to retrieve other Project-level information.

API.get_panel_project_name¶

get_panel_project_name(self)

Get the current panel's name.

Example:

from comet_ml import API

api = API()
project_name = api.get_panel_project_name()

The Project name could be useful in creating report-like Panels.

API.get_panel_report¶

get_panel_report(self)

Get a string detailing the number of hits the panel has made to the cloud.

API.get_panel_request_count¶

get_panel_request_count(self)

Get the number of requests made to the Comet cloud server.

API.get_panel_workspace¶

get_panel_workspace(self)

Get a list of workspace names.

Example:

from comet_ml import API

api = API()
workspace = api.get_panel_workspace()

The Workspace name could be useful in creating report-like Panels.

API.get_project¶

get_project(self, workspace=None, project_name=None, project_id=None)

Return the details of a project in a workspace.

Args:

workspace: String, the name of the workspace
project_name: String, the name of the project

Returns a dict of project details if the workspace/project exists, otherwise None.

Examples:

>>> api.get_project("workspace", "project-name")
{'projectId': 'project-id',
'projectName': 'project-name',
'ownerUserName': 'user name',
'projectDescription': 'my description',
'workspaceName': 'workspace',
'numberOfExperiments': 14,
'lastUpdated': 1571747775420,
'public': False}

>>> api.get_project("workspace", "non-existent-project-name")
None

API.get_project_columns¶

get_project_columns(self, workspace=None, project_name=None,
    project_id=None)

Given a workspace and project_name return the column names, types, etc.

API.get_project_experiments¶

get_project_experiments(self, workspace=None, project_name=None)

Return a dict of experiment_keys to metadata JSONs of the experiments in a project.

API.get_project_notes¶

get_project_notes(self, project_id)

Get the notes of a project.

Args:

project_id: String, the ID of the project

Returns: a string

Example:

>>> api.get_experiment_project_notes("my-workspace", "my-project")
"These are my project-level notes"

API.get_projects¶

get_projects(self, workspace)

Return the details of the projects in a workspace.

Args:

workspace: String, the name of the workspace

Returns a list of project details in workspace.

API.get_query_variables¶

get_query_variables(self, workspace=None, project_name=None,
    project_id=None)

Return the query variables of a project in a workspace. Used with API.query().

Args:

workspace: String, the name of the workspace
project_name: String, the name of the project

Returns objects used in forming queries, like:

[Metadata("user_name"),
Metadata("start_server_timestamp"),
Tag("my_tag"),
...]

API.get_workspaces¶

get_workspaces(self)

Return a list of names of the workspaces for this user.

API.query¶

query(self, workspace=None, project_name=None, query=None, archived=False)

Perform a query on a workspace/project to find matching APIExperiment. Queries are composed of

Args:

workspace: String, the name of the workspace
project_name: String, the name of the project
query: a query expression (see below)
archived: (optional boolean), query the archived experiments if True

((QUERY-VARIABLE OPERATOR VALUE) & ...)

# or:

(QUERY-VARIABLE.METHOD(VALUE) & ...)

where:

QUERY-VARIABLE is Environment(NAME), Metric(NAME), Parameter(NAME), Other(NAME), Metadata(NAME), or Tag(VALUE).

OPERATOR is any of the standard mathematical operators ==, <=, >=, !=, <, >.

METHOD is between(), contains(), startswith(), or endswith().

You may also place the bitwise ~ not operator in front of an expression which means to invert the expression. Use & to combine additional criteria. Currently, | (bitwise or) is not supported.

VALUE can be any query type, includeing string, boolean, double, datetime, or timenumber (number of seconds). None and "" are special values that mean NULL and EMPTY, respectively. Use API.get_query_variables(WORKSPACE, PROJECT_NAME) to see query variables and types for a project.

When using datetime, be aware that the backend is using UTC datetimes. If you do not receive the correct experiments via a datetime query, please check with the web UI query builder to verify timezone of the server.

query() returns a list of matching APIExperiments().

Examples:

# Find all experiments that have an acc metric value > .98:
>>> api.query("workspace", "project", Metric("acc") > .98)
[APIExperiment(), ...]

# Find all experiments that have a loss metric < .1 and
# a learning_rate parameter value >= 0.3:
>>> loss = Metric("loss")
>>> lr = Parameter("learning_rate")
>>> query = ((loss < .1) & (lr >= 0.3))
>>> api.query("workspace", "project", query)
[APIExperiment(), ...]

# Find all of the experiments tagged "My simple tag":
>>> tagged = Tag("My simple tag")
>>> api.query("workspace", "project", tagged)
[APIExperiment(), ...]

# Find all experiments started before Sept 24, 2019 at 5:00am:
>>> q = Metadata("start_server_timestamp") < datetime(2019, 9, 24, 5)
>>> api.query("workspace", "project", q)
[APIExperiment(), ...]

# Find all experiments lasting more that 2 minutes (in seconds):
>>> q = Metadata("duration") > (2 * 60)
>>> api.query("workspace", "project", q)
[APIExperiment(), ...]

Notes:

Use ~ for not on any expression
Use ~QUERY-VARIABLE.between(2,3) for values not between 2 and 3
Use (QUERY-VARIABLE == True) for truth
Use (QUERY-VARIABLE == False) for not true
Use (QUERY-VARIABLE == None) for testing null
Use (QUERY-VARIABLE != None) or ~(QUERY-VARIABLE == None) for testing not null
Use (QUERY-VARIABLE == "") for testing empty
Use (QUERY-VARIABLE != "") or ~(QUERY-VARIABLE == "") for testing not empty
Use Pythons datetime(YEAR, MONTH, DAY, HOUR, MINUTE, SECONDS) for comparing datetimes, likeMetadata("start_server_timestamp")orMetadata("end_server_timestamp")`
Use seconds for comparing timenumbers, like Metadata("duration")
Use API.get_query_variables(WORKSPACE, PROJECT_NAME) to see query variables and types.

Do not use and, or, not, is, or in. These are logical operators and you must use mathematical operators for queries. For example, always use == where you might usually use is.

API.query_project¶

query_project(self, workspace, project_name, predicates, archived=False)

Given a workspace, project_name, and predicates, return matching experiments.