Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.4.0
    • Component/s: Artifact Storage
    • Labels:
      None

      Description

      A sharded filestore with redundancy of R (should be >= 2 and <= Number of state aware file sub provider).

      • Each checksum file saved R times on different mount points.
      • On each upload the redundancy value of most free mount points are used.
      • Read event is looking for the file named sha1 using a rotating (round robin) list of mount point (no DB of where the file is).
      • At the end of every GC if R >= 2, the list of files presents on each filestore will be analyzed to make sure R is met for all files. If not, the file will be copied using the _pre + atomic move to another mount point based on quota management logic described above.

      A default configuration provided:

          <!-- Redundant Sharding with 2 shards default chain -->
          <chain template="redundant-shards">
              <provider id="cache-fs" type="cache-fs">
                  <provider id="redundant" type="redundant">
                      <sub-provider id="shard-state-aware-1" type="state-aware"/>
                      <sub-provider id="shard-state-aware-2" type="state-aware"/>
                  </provider>
              </provider>
          </chain>
          <!-- State Aware Shards FS provider configuration -->
          <provider id="shard-state-aware-1" type="state-aware">
              <dir>shard-state-aware-1</dir>
          </provider>
          <provider id="shard-state-aware-2" type="state-aware">
              <dir>shard-state-aware-2</dir>
          </provider>
      

      That can be activated by placing the following in (etc or ha-etc)/binarystore.xml:

      <config version="2">
          <chain template="redundant-shards"/>
      </config>
      

      Also more complicated setup can be achieved like:

      <config version="2">
          <chain>
              <provider id="cache-fs" type="cache-fs">
                <provider id="redundant" type="redundant">
                  <sub-provider id="shard-state-aware-1" type="state-aware"/>
                  <sub-provider id="shard-state-aware-2" type="state-aware"/>
                  <sub-provider id="shard-state-aware-3" type="state-aware"/>
                  <sub-provider id="shard-state-aware-4" type="state-aware"/>
                  <sub-provider id="shard-state-aware-5" type="state-aware"/>
                </provider>
              </provider>
          </chain>
          <provider id="redundant" type="redundant">
              <redundancy>3</redundancy>
          </provider>
          <provider id="shard-state-aware-1" type="state-aware">
              <dir>shard-state-aware-1</dir>
          </provider>
          <provider id="shard-state-aware-2" type="state-aware">
              <dir>shard-state-aware-2</dir>
          </provider>
          <provider id="shard-state-aware-3" type="state-aware">
              <dir>shard-state-aware-3</dir>
          </provider>
          <provider id="shard-state-aware-4" type="state-aware">
              <dir>shard-state-aware-4</dir>
          </provider>
          <provider id="shard-state-aware-5" type="state-aware">
              <dir>shard-state-aware-5</dir>
          </provider>
      </config>
      

        Attachments

          Activity

            People

            • Assignee:
              freds Fred Simon
              Reporter:
              freds Fred Simon
              Assigned QA:
              Dima Nevelev (Inactive)
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: