CLARIN Federated Content Search (CLARIN-FCS) - Core 1.0

The key words MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT, SHOULD, SHOULD NOT, RECOMMENDED, MAY, and OPTIONAL in this document are to be interpreted as in RFC2119.

The following typographic conventions for XML fragments will be used throughout this specification:

Endpoints and Clients MUST adhere to the XML-Namespaces specification. The CLARIN-FCS interface specification generally does not dictate whether XML elements should be serialized in their prefixed or non-prefixed syntax, but Endpoints MUST ensure that the correct XML namespace is used for elements and that XML namespaces are declared correctly. Clients MUST be agnostic regarding syntax for serializing the XML elements, i.e. if the prefixed or un-prefixed variant was used, and SHOULD operate solely on expanded names, i.e. pairs of namespace name and local name.

The following XML namespace names and prefixes are used throughout this specification. The column "Recommended Syntax" indicates which syntax variant SHOULD be used by the Endpoint to serialize the XML response.

Because CLARIN-FCS aims to integrate several heterogeneous Search Engines it needs to support the different features of the Search Engines, e.g. some only allow simple search while other support wild-cards or regular expressions. Also, the available resources have different properties, e.g. some resources provide Part-Of-Speech annotations, while others are transcripts of an audio signal or lexicographic resources consisting of entries. To accommodate for these different features, CLARIN-FCS defines several Capabilities. A Capability defines a certain feature that is part of CLARIN-FCS, e.g. what kind of queries are supported. Each Endpoint implements some (or all) of these Capabilities. The Endpoint will announce the capabilities it provides to allow a Client to auto-tune itself (see section Endpoint Description). Each Capability is identified by a Capability Identifier, which uses the URI syntax.

The following Capabilities are defined by CLARIN-FCS:

Endpoints MUST implement the Basic Search Capability.

Endpoints MUST NOT invent custom Capability Identifiers and MUST only use the values defined above.

The Search Engine will produce a result set containing several hits as the outcome of processing a query. The Endpoint MUST serialize these hits in the CLARIN-FCS result format. Endpoints are REQUIRED to adhere to the principle, that one hit MUST be serialized as one CLARIN-FCS result record and MUST NOT combine several hits in one CLARIN-FCS result record. E.g., if a query matches five different sentences within one text (= the resource), the Endpoint must serialize them as five SRU records each with one Hit each referencing the same containing Resource (see section Operation "searchRetrieve").

CLARIN-FCS uses a customized format for returning results. Resource and Resource Fragments serve as containers for hit results, which are presented in one or more Data View. The following section describes the Resource format and Data View format and section Operation "searchRetrieve" will describe, how hits are embedded within SRU responses.

Endpoints need to provide information about their capabilities to support auto-configuration of Clients. The Endpoint Description mechanism provides the necessary facility to provide this information to the Clients. Endpoints MUST encode their capabilities using an XML format and embed this information into the SRU/CQL protocol as described in section Operation "explain". The XML fragment generated by the Endpoint for the Endpoint Description MUST be valid according to the XML schema "Endpoint-Description.xsd".

The XML fragment for Endpoint Description is encoded as an <ed:EndpointDescription> element, that contains the following attributes and children:

The <ed:Resource> element contains a basic description of a resource that is available at the Endpoint. A resource is a searchable entity, e.g. a single corpus. The <ed:Resources> has a mandatory @pid attribute that contains persistent identifier of the resource. This value MUST be the same as the MdSelfLink of the CMDI record describing the resource. The <ed:Resources> element contains the following children:

Example simple Endpoint Description shows a simple Endpoint Description for an Endpoint that only supports the Basic Search Capability and only provides the Generic Hits Data View, which is indicated by a <ed:SupportedDataView> element. This element carries an @id attribute with a value of hits, the recommended value for the short identifier, and indicates a delivery policy of send-by-default by the @delivery-policy attribute. It only provides one top-level resource identified by the persistent identifier http://hdl.handle.net/4711/0815. The resource has a title as well as a description in German and English. A landing page is located at http://repos.example.org/corpus1.html. The predominant language in the resource contents is German. Only the Generic Hits Data View is supported for this resource, because the <ed:AvailableDataViews> element only references the <ed:SupporedDataView> element with the @id with a value of hits.

The more complex Example CMDI Endpoint Description show an Endpoint Description for an Endpoint that, similar to Example simple Endpoint Description, supports the Basic Search capability. In addition to the Generic Hits Data View, it also supports the CMDI Data View. The delivery polices are send-by-default for the Generic Hits Data View and need-to-request for the CMDI Data View. The Endpoint has two top-level resources (identified by the persistent identifiers http://hdl.handle.net/4711/0815 and http://hdl.handle.net/4711/0816. The second top-level resource has two independently searchable sub-resources, identified by the persistent identifier http://hdl.handle.net/4711/0816-1 and http://hdl.handle.net/4711/0816-2. All resources are described using several properties, like title, description, etc. The first top-level resource provides only the Generic Hits Data View, while the other top-level resource including its children provide the Generic Hits and the CMDI Data Views.

Endpoints can add custom extensions, i.e. custom data, to the Result Format. This extension mechanism can for example be used to provide hints for an (XSLT/XQuery) application that works directly on CLARIN-FCS, e.g. to allow it to generate back and forward links to navigate in a result set.

An Endpoint MAY add arbitrary XML fragments to the extension hooks provided in the <fcs:Resource> element (see the XML schema for "Resource.xsd"). The XML fragment for the extension MUST use a custom XML namespace name for the extension. Endpoints MUST NOT use XML namespace names that start with the prefixes http://clarin.eu, http://www.clarin.eu/, https://clarin.eu or https://www.clarin.eu/.

A Client MUST ignore any custom extensions it does not understand and skip over these XML fragments when parsing the Endpoint’s response.

The appendix contains an example, how an extension could be implemented.

SRU (Search/Retrieve via URL) specifies a general communication protocol for searching and retrieving records and the CQL (Contextual Query Language) specifies an extensible query language. CLARIN-FCS is built on SRU 1.2. A subsequent specification may be built on SRU 2.0.

Endpoints and Clients MUST implement the SRU/CQL protocol suite as defined in OASIS-SRU-Overview, OASIS-SRU-APD, OASIS-CQL, SRU-Explain, SRU-Scan, especially with respect to:

Endpoints and Clients MUST implement the APD Binding for SRU 1.2, as defined in OASIS-SRU12. Endpoints and Clients MAY also implement APD binding for version 1.1 or version 2.0.

Endpoints and Clients MUST use the following XML namespace names (namespace URIs) for serializing responses:

CLARIN-FCS deviates from the OASIS specification OASIS-SRU-Overview and OASIS-SRU12 to ensure backwards comparability with SRU 1.2 services as they were defined by the LOC-SRU12.

Endpoints or Clients MUST support CQL conformance Level 2 (as defined in OASIS-CQL, section 6), i.e. be able to parse (Endpoints) or serialize (Clients) all of CQL and respond with appropriate error messages to the search/retrieve protocol interface.

Endpoints MUST generate diagnostics according to OASIS-SRU12, Appendix C for error conditions or to indicate unsupported features. Unfortunately, the OASIS specification does not provides a comprehensive list of diagnostics for CQL-related errors. Therefore, Endpoints MUST use diagnostics from LOC-DIAG, section "Diagnostics Relating to CQL" for CQL related errors.

Endpoints MUST support the HTTP GET OASIS-SRU12, Appendix B.1 and HTTP POST OASIS-SRU12, Appendix B.2 lower level protocol binding. Endpoints MAY also support the SOAP OASIS-SRU12, Appendix B.3 binding.

The explain operation of the SRU protocol serves to announce server capabilities and to allow clients to configure themselves automatically. This operation is used similarly.

The Endpoint MUST respond to a explain request by a proper explain response. As per SRU-Explain, the response MUST contain one <sru:record> element that contains an SRU Explain record. The <sru:recordSchema> element MUST contain the literal http://explain.z3950.org/dtd/2.0/, i.e. the official identifier for Explain records.

According to the Capabilities supported by the Endpoint the Explain record MUST contain the following elements:

To support auto-configuration in CLARIN-FCS, the Endpoint MUST provide an Endpoint Description. The Endpoint Description is included in explain response utilizing SRUs extension mechanism, i.e. by embedding an XML fragment into the <sru:extraResponseData> element. The Endpoint MUST include the Endpoint Description only if the Client performs an explain request with the extra request parameter x-fcs-endpoint-description with a value of true. If the Client performs an explain request without supplying this extra request parameter the Endpoint MUST NOT` include the Endpoint Description. The format of the Endpoint Description XML fragment is defined in Endpoint Description.

The following example shows a request and response to an explain request with added extra request parameter x-fcs-endpoint-description:

The scan operation of the SRU protocol is currently not used in the Basic Search capability of CLARIN-FCS. Future capabilities may use this operation, therefore it NOT RECOMMENDED for Endpoints to define custom extensions that use this operation.

The searchRetrieve operation of the SRU protocol is used for searching in the Resources that are provided by the Endpoint. The SRU protocol defines the serialization of request and response formats in OASIS-SRU12. In SRU, search result hits are encoded down to a record level, i.e. the <sru:record> element, and SRU allows records to be serialized in various formats, so called record schemas.

Endpoints MUST support the CLARIN-FCS record schema (see section Result Format) and MUST use the value http://clarin.eu/fcs/resource for the responseItemType ("record schema identifier"). Endpoints MUST represent exactly one hit within the Resource as one SRU record, i.e. <sru:record> element.

The following example shows a request and response to a searchRetrieve request with a term-only query for "cat":

In general, the Endpoint is REQUIRED to accept an unrestricted search and SHOULD perform the search operation on all Resources that are available at the Endpoint. If that is for some reason not feasible, e.g. performing an unrestricted search would allocate too many resources, the Endpoint MAY independently restrict the search to a scope that it can handle. If it does so, it MUST issue a non-fatal diagnostics http://clarin.eu/fcs/diagnostic/2 ("Resource set too large. Query context automatically adjusted."). The details field of diagnostics MUST contain the persistent identifier of the resources to which the query scope was limited to. If the Endpoint limits the query scope to more than one resource, it MUST generate a separate non-fatal diagnostic http://clarin.eu/fcs/diagnostic/2 for each of the resources.

The Client can request the Endpoint to restrict the search to a sub-resource of these Resources. In this case, the Client MUST pass a comma-separated list of persistent identifiers in the x-fcs-context extra request parameter of the searchRetrieve request. The Endpoint MUST then restrict the search to those Resources, which are identified by the persistent identifiers passed by the Client. If a Client requests too many resources for the Endpoint to handle with x-fcs-context, the Endpoint MAY issue a fatal diagnostic http://clarin.eu/fcs/diagnostic/3 ("Resource set too large. Cannot perform Query.") and terminate processing. Alternatively, the Endpoint MAY also automatically adjust the scope and issue a non-fatal diagnostic http://clarin.eu/fcs/diagnostic/2 (see above). And Endpoint MUST NOT issue a http://clarin.eu/fcs/diagnostic/3 diagnostic in response to a request, if a Client performed the request without the x-fcs-context extra request parameter.

The Client can extract all valid persistent identifiers from the @pid attribute of the <ed:Resource> element, obtained by the explain request (see section Operation "explain" and section Endpoint Description). The list of persistent identifiers can get extensive, but a Client can use the HTTP POST method instead of HTTP GET method for submitting the request.

For example, to restrict the search to the Resource with the persistent identifier http://hdl.handle.net/4711/0815 the Client must issue the following request:

+

To restrict the search to the Resources with the persistent identifier http://hdl.handle.net/4711/0815 and http://hdl.handle.net/4711/0816-2 the Client must issue the following request:

+

If an invalid persistent identifier is passed by the Client, the Endpoint MUST issue a http://clarin.eu/fcs/diagnostic/1 diagnostic, i.e. add the appropriate XML fragment to the <sru:diagnostics> element of the response. The Endpoint MAY treat this condition as fatal, i.e. just issue the diagnostic and perform no search, or it MAY treat it as non-fatal and perform the search.

If a Client wants to request one or more Data Views, that are handled by Endpoint with the need-to-request delivery policy, it MUST pass a comma-separated list of Data View identifier in the x-fcs-dataviews extra request parameter of the 'searchRetrieve' request. A Client can extract valid values for the Data View identifiers from the @id attribute of the <ed:SupportedDataView> elements in the Endpoint Description of the Endpoint (see section Operation "explain" and section Endpoint Description).

For example, to request the CMDI Data View from an Endpoint that has an Endpoint Description, as described in Example of simple Endpoint Description with optional CMDI Data View, a Client would need to use the Data View identifier cmdi and submit the following request:

+

If an invalid Data View identifier is passed by the Client, the Endpoint MUST issue a http://clarin.eu/fcs/diagnostic/4 diagnostic, i.e. add the appropriate XML fragment to the <sru:diagnostics> element of the response. The Endpoint MAY treat this condition as fatal, i.e. simply issue the diagnostic and perform no search, or it MAY treat it a non-fatal and perform the search.

The following extra request parameters are used in CLARIN-FCS. The column SRU operations lists the SRU operation, for which this extra request parameter is to be used. Clients MUST NOT use the parameter for an operation that is not listed in this column. However, if a Client sends an invalid parameter, an Endpoint SHOULD issue a fatal diagnostic "Unsupported Parameter" (info:srw/diagnostic/1/8) and stop processing the request. Alternatively, an Endpoint MAY silently ignore the invalid parameter.

Apart from the SRU diagnostics defined in OASIS-SRU12, Appendix C and LOC-DIAG, the following diagnostics are used in CLARIN-FCS. The "Details Format" column specifies what SHOULD be returned in the details field. If this column is blank, the format is "undefined" and the Endpoint MAY return whatever it feels appropriate, including nothing.

Persistent Identifiers from the Handle system are defined in two syntax variants: a regular URI format for the Handle protocol, i.e. with a hdl: prefix, or actionable URIs with a http://hdl.handle.net/ prefix. Generally, CLARIN software should support both syntax variants, therefore the CLARIN-FCS Interface Specification does not endorse a specific syntax variant. However, Endpoints are recommended to use the actionable syntax variant.

Centers are encouraged to provide links to their CLARIN-FCS Endpoints in the metadata records for their resources. Other services, like the VLO, can use this information for automatically configuring an Aggregator for searching resources at the Endpoint. To refer to an Endpoint, a <cmdi:ResourceProxy> element with child-element <cmdi:ResourceType> set to the value SearchService and a @mimetype attribute with a value of application/sru+xml need to be added to the CMDI record. The content of the <cmdi:ResourceRef> element must contain a URI that points to the Endpoint web service.

The CLARIN-FCS protocol specification allows Endpoints to add custom data to their responses, e.g. to provide hints to an (XSLT/XQuery) application that works directly on CLARIN-FCS. It could use the custom data to generate back and forward links for a GUI to navigate in a result set.

The following example illustrates how extensions can be embedded into the Result Format:

An Aggregator can use the @ref attributes of the <fcs:Resource>, <fcs:ResourceFragment> or <fcs:DataView> elements to provide a link for the user to directly jump to the resource at a Repository. To support hit highlighting, an Endpoint can augment the URI in the @ref attribute with query parameters to implement hit highlighting in the Repository.

In the following example, the URI http://repos.example.org/resource.cgi/<pid> is a CGI script that displays a given resource at the Repository in HTML format and uses the highlight query parameter to add highlights to the resource. Of course, it’s up to the Endpoint and the Repository, if and how they implement such a feature.